#metabrainz

ok, good

it's going to take a few mins to complete the rest because we have to update apt package cache and then install the new package

also restarting the service takes some time for some reason

i think consul doesn't signal success to systemd until various internal checks have passed

likely

new log message 'EventMemberUpdate' https://www.irccloud.com/pastebin/BTEYe0bp/

clash done

yeah, i think that when that message is logged is when the service restart completes

~90 seconds later

suspicious that it seems to take exactly 1m30

yeah

i think we should set serial to 2 or 3 when upgrading clients in future

this is going to take a little while

2 minutes per server, but we could do them in parallel

well maybe 2 or 3

for clients, I don't think that's a problem

got a failed got gaga and then a join

*for gaga

yes, I saw that

same for hip

messages are obscure...

Go team go ! You both get a consulation prize at the end of this, right?

i have beers in the fridge

(that wasn't a typo)

kiss load is quite high

sorry monkey, sense of humour bypass

:D

Understandable

quite high is an understatement

ah, just dropped

was it due to a consul service causing reloads / restarts?

time is matching

but the load increases started 15 minutes sooner

4 left

we don't have EventMemberUpdate each time...

when that happens the service restarts almost instantly

Serge did 2 rounds https://www.irccloud.com/pastebin/OgrELJlq/

we're going to get an update from zappa

then it's done

ok, I think we can restore services now

ok, done

thank god for that

ready to go?

yep

ok, everything's back

atj: please commit playbooks you used for the upgrade

Should I tweet we're back online?

if everyone is happy things are working

I checked BB, LB and CB web pages, working 👍

LB API also responding

we have few alerts still on, but most should resolve soon

MB website and WS look ok

I merged https://github.com/metabrainz/docker-server-con... so please be careful next time you run scripts from it

I can push consul config changes to docker-server-configs now, right?

bitmap: yes

thanks

Tweeted for BB, CB and LB.

atj: we have to go through servers to remove old containers: dnsmasq, consulagent

atj: you can merge the consul-migration branch now

zas: i'm just re-arranging and amending some commits

!m atj and zas 👏

You're doing good work, atj and zas 👏!

https://www.instagram.com/reel/CpiPEALDEyC/?igs... monkey, mayhem, can’t we have stuff like this on LB in terms of stats and cool things to explore?

I mean, people dig such facts. The likes, shares and comments on the post can tell

This is a good way to make noise about our stats

However, I think MB would play a bigger role in all this

atj: I'll clean old containers

zas: ok thanks

zas: updated PR - https://github.com/metabrainz/metabrainz-ansibl...

just have another quick look through and confirm you're happy

bitmap: test. is still b0rked, do you know what the issue is? i think Pratha-Fish was hoping to use it for his GSoC development

zas, atj, bitmap: can I release MBS now? :)

atj: Pratha-Fish: yeah I’m importing a new test DB now, will let you know when it’s ready

reosarevok: seems so

Ok :)

reosarevok: yes, but beware running scripts from docker-server-configs

Oh. Still cannot do that?

you can, but carefully

The upgrade system checks stuff in d-s-c

Oh

because I made a lot of changes

Well only one way to find out if it works I guess

I'll do beta first and hope for the best

bitmap: wanna merge https://github.com/metabrainz/musicbrainz-serve... ? :)

[musicbrainz-server] 14mwiencek merged pull request #2962 (03beta…mbs-13120): MBS-13120: Fix note-too-old check for modifications https://github.com/metabrainz/musicbrainz-serve...

I guess I'll skip translations this time since yvanzo hasn't mentioned being done with weblate

bitmap: thanks

Trying a beta update

Guess not trivial

That'd be https://github.com/metabrainz/musicbrainz-serve...

I assume on

So set_service_maintenance - zas, maybe https://github.com/metabrainz/docker-server-con... didn't work?

Or maybe something really is not running, but

mayhem epic. I"ll reach out to you soon about troi. Gotta put that degree in use somewhere 🗿

reosarevok: can you run bash -x ./script/update_containers.sh beta

Sure can try

reosarevok: bitmap hey thanks for the quick diagnosis, but it's alright if you take your time. I am unfortunately out of station for today and tomorrow 🥲. Please take your time fixing it

atj: I guess it's safe to paste the log here? :)

not sure

I'll paste in pm in case then

zas: there's an issue here: https://github.com/metabrainz/musicbrainz-serve...

The running_image assignment is getting an id instead of the name that I assume was getting before

$running_image is a container id, but $updated_image is a container name

Is there a reason why the result of docker ps --filter "name=$container" --format '{{.Image}}' would have changed?

nope, but if you want the name that's not .Image

those are running on paco

docker ps --filter name=musicbrainz-webservice-beta --format '{{.Names}}'

kinda of redundant tho i guess L()

but if you filter by name ... I guess you have it

reosarevok: what do you want to achieve?

i guess it's to check if the docker image matches?

which would need the ID rather than the name

This is not my code, but yeah, the idea is to check if the image needs updating or not

yes, who wrote this code? i thought it was zas :P

Hi theflash_ !

not at all

yvanzo:

* without the :

Can you please go to your local codebase on xcode, click on the artist model file, add it to Target Membership from the right panel and then push that code to github? theflash_

Basically, it checks whether the new image we have is the same as the one already running, if it is it does nothing, if it is not it puts the container in maintenance mode and then updates it, then puts it out of maintenance mode

Don't ask me what maintenance mode did, I don't remember

also the filter should probably be stricter to avoid issues

that's a regex

--filter "name=^musicbrainz-webservice-beta$"

lucifer: there's still an alert on https://stats.metabrainz.org/d/000000079/alerts...

about spotify_metadata

atj: do you have the change to hold the docker version in Ansible ready? I don't think you committed it

[listenbrainz-server] 14amCap1712 merged pull request #2499 (03master…Fix-uses-of-APIUnauthorized-error): LB-1256 Fix uses of APIUnauthorized error https://github.com/metabrainz/listenbrainz-serv...

[listenbrainz-server] 14amCap1712 merged pull request #2490 (03master…dependabot/pip/requests-2.31.0): Bump requests from 2.27.1 to 2.31.0 https://github.com/metabrainz/listenbrainz-serv...

[listenbrainz-server] 14amCap1712 opened pull request #2502 (03master…spotify-cache-fix): Fix spotify-metadata-cache updater https://github.com/metabrainz/listenbrainz-serv...

Ok, I'll talk about it with yvanzo in the morning :)

🐟

🗿