not sure if the container needs to be removed for sir, if you already updated it then I guess noot
2021-05-17 13740, 2021
reosarevok
I'll tweet once more
2021-05-17 13701, 2021
bitmap
aight. still waiting for the consul changes to propagate
2021-05-17 13702, 2021
bitmap
I guess it has, since the editing pages say so. thought we had a banner for that
2021-05-17 13710, 2021
bitmap
weird that it's not showing up
2021-05-17 13733, 2021
_lucifer
its showing form e
2021-05-17 13742, 2021
_lucifer
>MusicBrainz will be taken down temporarily today at 17 UTC for our schema change upgrade! It can take some time to backup and vacuum the database, but we hope to only take about an hour. Thank you for your patience.
2021-05-17 13749, 2021
reosarevok
That's the manual banner
2021-05-17 13756, 2021
_lucifer
ah ok
2021-05-17 13756, 2021
bitmap
I'll start the backup
2021-05-17 13757, 2021
reosarevok
But there should be a "we're in RO" banner too IIRC
dump is on the url table, so hopefully noot that many left
2021-05-17 13700, 2021
bitmap
I guess it was a bit slower with all the RO traffic still
2021-05-17 13709, 2021
bitmap
ok it's done! I'll bring services down in docker-server-configs
2021-05-17 13727, 2021
bitmap
yvanzo: reosarevok: can you help bring MBS containers down?
2021-05-17 13757, 2021
bitmap
I'll do zappa & serge, you can do the other six
2021-05-17 13709, 2021
reosarevok
Ok
2021-05-17 13713, 2021
bitmap
just the website and webservice containers
2021-05-17 13717, 2021
bitmap
leave the production-cron one
2021-05-17 13717, 2021
ruaok is available as well
2021-05-17 13725, 2021
reosarevok
I'll take burnside cage hip
2021-05-17 13755, 2021
reosarevok
Just run docker stop musicbrainz-website-prod musicbrainz-webservice-prod && docker rm musicbrainz-website-prod musicbrainz-webservice-prod
2021-05-17 13757, 2021
reosarevok
?
2021-05-17 13718, 2021
reosarevok
Seems easy
2021-05-17 13726, 2021
bitmap
yup
2021-05-17 13727, 2021
reosarevok
I was already in hip, starting there
2021-05-17 13722, 2021
akshaaatt[m]
<ruaok "GSoC announcement mails are arri"> Amazing. Thanks for the opportunity. Happy to join the team 😊 Hope to stay connected for a long long time moving forward. Thanks again everyone!
I assume that is fine. Very informational return messages
2021-05-17 13738, 2021
ruaok
akshaaatt[m]: :)
2021-05-17 13745, 2021
_lucifer
congrats akshaaatt[m]! :-D
2021-05-17 13738, 2021
reosarevok
burnside now
2021-05-17 13715, 2021
yvanzo
bitmap: sitemapstoo?
2021-05-17 13729, 2021
reosarevok
burnside down, cage down
2021-05-17 13734, 2021
bitmap
I think I already removed sitemaps
2021-05-17 13738, 2021
bitmap
on yehudi
2021-05-17 13703, 2021
reosarevok
Taking trille
2021-05-17 13708, 2021
adhi001 joined the channel
2021-05-17 13730, 2021
reosarevok
pink
2021-05-17 13736, 2021
reosarevok
And ludwig
2021-05-17 13744, 2021
reosarevok
bitmap: did you take down prod in zappa too?
2021-05-17 13751, 2021
bitmap
yeah
2021-05-17 13705, 2021
reosarevok
Then they should be all down
2021-05-17 13710, 2021
reosarevok
Do we have a way to check? :p
2021-05-17 13709, 2021
bitmap
I'm watching pg_stat_activity :)
2021-05-17 13717, 2021
_lucifer
ruaok, is there a way to speed up downloads in spark cluster? its slowwwwwww.........
2021-05-17 13722, 2021
bitmap
all the remaining connections are telegraf so I think we're good
2021-05-17 13752, 2021
_lucifer
but i guess the dump is largeeeeeee that's why
2021-05-17 13758, 2021
bitmap
I'll generate the replication packet now
2021-05-17 13756, 2021
bitmap
replication-140655.tar.bz2
2021-05-17 13726, 2021
bitmap
it's in /var/lib/docker/volumes/musicbrainz-replication-packets/_data/replication-140655.tar.bz2
2021-05-17 13701, 2021
bitmap
moving onto upgrade.sh then!
2021-05-17 13716, 2021
bitmap
oh, dropping sir triggers first
2021-05-17 13748, 2021
mglubb has quit
2021-05-17 13711, 2021
bitmap
done
2021-05-17 13716, 2021
bitmap
going to start upgrade.sh, fingers crossed
2021-05-17 13718, 2021
yvanzo
Reloaded Solr collections.
2021-05-17 13731, 2021
yvanzo
Updated image for sir-prod.
2021-05-17 13702, 2021
bitmap
Mon May 17 17:59:34 UTC 2021 : Vacuuming DB.
2021-05-17 13710, 2021
bitmap
schema changes seem to have applied ook
2021-05-17 13703, 2021
ruaok
:D
2021-05-17 13727, 2021
reosarevok
Ook! Ook ook ook!
2021-05-17 13740, 2021
yvanzo
reosarevok: other way to check (same to stop/rm): for service in musicbrainz_webservice musicbrainz_website; do for deploy_env in prod beta; do for host in $(./scripts/list_nodes.sh $service $deploy_env); do ssh $host docker ps --filter name=${service/_/-}-$deploy_env; done; done; done
2021-05-17 13735, 2021
bitmap
still vacuuming ofcourse
2021-05-17 13703, 2021
ruaok
> [Alerting] PG WAL count
2021-05-17 13709, 2021
ruaok
on telegram. FYI
2021-05-17 13701, 2021
bitmap
hrm yeah that's skyrocketed. but I guess it's normal until we do another barman backup
2021-05-17 13728, 2021
bitmap
didn't account for the number of wal changes the schema change would produce
2021-05-17 13749, 2021
bitmap
vacuuming just finished, now I just have to build the new materialized tables
2021-05-17 13706, 2021
zas
!m bitmap
2021-05-17 13706, 2021
BrainzBot
You're doing good work, bitmap!
2021-05-17 13751, 2021
akashgp09 joined the channel
2021-05-17 13754, 2021
bitmap
this might take just as long as vacuuming or a bit longer unfortunately
iliekcomputers: Hi! I don't have write acces to that repo, could you please look at this PR and merge if it looks good to you? Build fine for me locally: https://github.com/metabrainz/docker-node/pull/1
2021-05-17 13736, 2021
yvanzo
bitmap, reosarevok: in your mbsssss copy, run: git tag -d v-2021-05-14 # locally remove your local tag as I overwrote it
2021-05-17 13739, 2021
bitmap
ok
2021-05-17 13739, 2021
iliekcomputers
Mr_Monkey: I added metabrainz/coreteam to the repo, does that give you access now?
2021-05-17 13745, 2021
iliekcomputers
Mr_Monkey: I added metabrainz/coreteam to the repo, does that give you access now?
2021-05-17 13712, 2021
iliekcomputers
sorry, internet
2021-05-17 13714, 2021
iliekcomputers
:D
2021-05-17 13749, 2021
Mr_Monkey
Looks like it, thanks Param !
2021-05-17 13729, 2021
ShivamAwasthi has quit
2021-05-17 13717, 2021
yvanzo
bitmap: last replication packet with schema 25 will be #140655, right?
2021-05-17 13732, 2021
bitmap
yup
2021-05-17 13729, 2021
yvanzo
adding to upgrade instructions
2021-05-17 13718, 2021
yvanzo
(actually not needed)
2021-05-17 13712, 2021
bitmap
the script failed (fixed it) and I had to restart it, now it's taking forever. possibly stuck on some lock
2021-05-17 13728, 2021
bitmap
sigh
2021-05-17 13731, 2021
reosarevok
Oh no :(
2021-05-17 13757, 2021
bitmap
I killed all queries and restarted it, now it doesn't appear to be waiting anymore
