#metabrainz

yesterday's went until 19:23 so we'll surely have to stop them

bitmap: I rearranged a bit the post release sequence to quickly produce a sample data dump at first, as it can help with debugging Docker Compose if something goes wrong during the last tests.

ok, makes sense

Maybe we can even start the live data feed before starting a full data dump?

yvanzo: I tweeted, wanna post elsewhere since it's probably faster for you given you have accessed them already? :)

yvanzo: we could do it in that order too, yes (it just requires waiting around for the packet to be generated)

bitmap: I'm going to stop cron for sitemaps then

No need to stop the process in any specific way in advance, is there?

oh I already stopped the cron there, but I mean we can forcefully stop the container later

Oh

stopping cron doesn't affect existing jobs that are running

Sure, ok, I meant stop and remove the container :)

Sorry

I guess there's no reason to wait if we're reasonably sure it won't finish on time?

I just killed them

Thanks

(you can run the command to stop and remove the container though)

Done

yvanzo: are you waiting for a specific time for "Check Weblate repository status" (again)? :)

bitmap: I reordered the steps and made it clear that we should keep going on.

👍

reosarevok: Weblate just needed time to catch up with changes: It had “33 missing in the push branch”, now 0.

we can start the sample dump concurrently with the MB production cron

lucifer: it's soon time "to stop LB cron jobs that depend on the MB DB"

Ok, wasn't sure if it might be delaying the first packet by a while.

reosarevok: sure can do it right now

reosarevok: had you started the pg_dump of musicbrainz_db already?

Nope

But I can

oh we probably should have had that first in the list

Oops.

started it now

How long should this take?

reosarevok: It seems that I’m not receiving any confirmation mail from Twitter, so I cannot see what you posted and use it for other networks.

I just quoted the previous tweet with "Downtime in around an hour now. Finish your editing and wish us a quick release! - reo"

Not really around an hour anymore tho

the dump I made on the 7th is 27GB, so if it dumps at least 1GB per minute, hopefully under 30 minutes :)

it's at 2GB after 2 minutes...

(I moved it to the top of the list)

Thanks

reosarevok: Done.

bitmap, reosarevok: I also backed up MB web container logs just in case.

thanks, where are the backups?

In my home directory.

👍

the dump won't finish by 17 utc

I’m stopping SIR instances on rakim.

should we skip it or wait? this particular step was before we had barman and zfs snapshots

bitmap: which dump?

the aretha one I started a while ago, on edit_data now

the pg_dump of musicbrainz_db you started at 16:27

pg_dump is very easy to use, but if you are comfortable enough to restore with other methods, feel free to skip.

Oh, I thought you said 30 min or so?

How long do you think it'll take? We can wait 5 min extra, but

probably around 45 minutes in actuality, that was an early estimate

It’s also okay to wait a bit more if needed.

I think that's fine tbh

We can stop it if it seems like it'll take a lot longer than that by 17utc

but could be up to an hour, not 100% sure it will stay at the same rate

if something goes horribly wrong I assume we'd start with the zfs snapshot since it's taken right before the upgrade

That gives us a bit more time to gather restore instructions with those other methods then.

it's almost half done, so, probably fine to wait a little

👍

Finished edit_data, at least

did you take the snapshot or have you not reached that point?

not reached that point

if something was to wrong and you wanted to revert, it'd simply be a case of running `zfs rollback rpool/ROOT/ubuntu/srv/postgresql@preupgrade` and `zfs rollback rpool/ROOT/ubuntu/srv/postgresql/wal-12@preupgrade`

should be pretty much instantaneous and you'd be exactly where you were before you started

if anyone needs any help, shout@

hlep!

oh, not what you were going for, mayhem, sorry

atj: Ah, the wonders of ZFS snapshots. :-)

The dump is on editor, hopefully recording/release won't take forever

12:15 is the new downtime ETA

sorry, 17:15

12:15 am ;)

haha

👀

Thanks atj, I’ve added these two commands to the roadmap docs.

I’m focused on mirrors’ code now.

https://docs.google.com/document/d/1o8h1z02YBWs... should be public (read-only) for others too

dump is on track now

So hopefully it's on track

🤣

But when it will be on release it won’t be finished still

Actually, we have a second full mirror on wolf to test upgrading. :)

hi MatthewGlubb! We didn't break anything yet surely?

lol

Haha! Hi reosarevok! No. Just checking in to see that all systems are nominal for launch. I haven't looked, but I wondered if mb-solr related changes have been tagged in git to make it easy for me to know what to upgrade?

(we are already running Solr 9, so should be simple for us)

We're not releasing all the search stuff today in the end since it's a bit behind in testing but yvanzo can give you all the details :)

reosarevok: finished?

MatthewGlubb: 👍 even though we won’t provide officially support for it until we have deployed it in production with dumps.

bitmap: just did, yes!

Let's move on! :L)

* :)

nice, just on time

merging https://github.com/metabrainz/docker-server-con... then

yvanzo: are you done with all the image building now?

No worries. I can just rebuild our indices for Solr 9 with what I've already got. I'll leave you all be. Good luck!

reosarevok: No, I’m still at building images for test mirrors

I'll stop the PG things, can you do the rest?

MatthewGlubb: Sure, it works the same for SIR and MBS anyway :)

bitmap: did you do the zfs now?

no not yet

it's after the reboot

Oh, ok

zas: you can reboot jimmy and hendrix

bitmap: I'm stopping prod and beta

ok

are there any other hosts with non-redundant services we should prioritize rebooting? patton (MB redis store) comes to mind

rebooting hendrix

yes, any server we don't usually reboot

you can reboot patton too then

rebooting jimmy

Don't do patton yet

I'm still stopping services

k

I mean, unless it doesn't matter

reosarevok: tell me when

bitmap: had you stopped the artwork indexer?

Error response from daemon: No such container: artwork-indexer

I don't think so

the command is wrong, should be artwork-indexer-prod

fixed it in the doc

still waiting hendrix & jimmy to come back

reosarevok: I stopped it

So did I, heh

bitmap: hendrix is back

I guess I don't have all the latest ssh configs

bitmap: can you run that one?

bitmap: jimmy is back

Should have updated that earlier, oh well

ok

reosarevok: tell me when you're done with patton

bitmap: I took prod-cron down again

thanks

zas: I dunno if sir-prod runs on there, if it does, once bitmap stops it, if not, now

it doesn't

I'll proceed with the PG v16 upgrade

Ok, zas, feel free to reboot then :)

(sir-prod is stopped now)

rebooting patton

pg upgrade is running

patton's back