#metabrainz

alastairp: which copy of bono-data-set hoster is live on bono right now? your new PR branch? I have one tiny update to make (table name change for year lookup).

and then sentry fixes too. :)

no, I didn't release anything

thanks for that!

ah, those sentry alerts are from testing, then?

correct

ok, kewl.

let me just commit the table name change to main and deploy an update.

btw, I'm putting LB unit and int tests on jenkins, so that they don't take hours to run

+10000

then I'll get a PR for bono-datasets and LB to use the new version of datasethoster+sentry

after I pay people, I'll jump back into troi and then playlist serializing... I had some questions for you...

acrp-year-lookup moved to new mbid-mapping PR table name

uh oh... questions

yvanzo: hi, were we talking some months ago about removing test images from williams after running jenkins jobs?

MFW I realise it's now December.

Mr_Monkey: what do you mean? It's still March

Right?

we're going to wake up any time now

Mr_Monkey: but that means your holidays are close!

Wait, we don't have official holidays. Whoops.

You can meet with family and have a lovely time!

Wait, COVID restrictions. Whoops.

Ok, yeah, why is it December already

Well, less time until a vaccine I guess, hopefully. *that* is a positive, yes?

alerts due to critical upgrades, unexpected :(

Ok so the 502s are on your radar

ugh. need help zas?

not really yet, but I expect mess in our services

MB is already 502ing

Ah - glad that you are already aware of the 502s - I was just going to mention them.

Do you expect this to last? Should I tweet?

that has indeed arrive, zas.

a package upgrade shouldn't have restarted services, but it did :(

oy.

let's see what recovers or not

nothing yet. :(

should things be recovering now?

I guess so, but ...

smells like postgres.

zas: check pg bouncer.

did it come back ok?

Yeah, just visiting it now

perhaps restart it?

"Unable to acquire a connection"

Is everyhing else back up?

Mr_Monkey: yes, you're providing a good clue that none of our other sites are.

Mr_Monkey: not yet.

OK

wait a bit, it seems things are coming back after the storm, but we will find corpses....

LB won't come back up.

Yeah, my local ssh tunnel to pink works but there's no DB connection

acousticbrainz.org is unaffected (or already back up)

zas: how is pg bouncer?

So postgres-pink is running, but psql fails

Mmm, spoke too fast. Landing page *seems* unaffected only

yep, can you please restart pg_bouncer, zas?

confirmed.

DB is not up.

or rather DB is unreachable. pg bouncer likely.

[bookbrainz-site] MonkeyDo merged pull request #538 (master…typescript): Typescript - Stage 2 https://github.com/bookbrainz/bookbrainz-site/p...

pgbouncer doesn't start

wtf

which machine?

pink

zas: postgres on pink doesn't seem to have started correctly.

on floyd

floyd is main db

ah

I don't get why it doesn't start

I'll tweet for now just in case

yes please

Same

zas: postgres on floyd is NOT up.

pg bouncer cannot connect because the db is donw.

"Sorry about the 502 errors on MusicBrainz right now! Our database didn't come back up as expected after an upgrade (that we weren't expecting would take it down in the first place). We're working on it! - reo"

nope, and I don't know why

Sounds about right?

reosarevok: yup

zas: let me try and restart it.

Sent

ruaok: ok

yerp, not starting.

fun,

ruaok: let's try a full reboot in case of

think that will change anything?

we can try....

its not evening trying to start pg. wtaf?

well, yd cron[24]: (CRON) INFO (Skipping @reboot jobs -- not system startup)

ok, go for it zas

I'm not hopeful.

can you ping bitmap? we may need him

what a mess

pinged.

does it normally take this long to reboot floyd?

this is not promising at all. :(

it took time

but it's back

ok, not a reboot issue.

though pg container still doesn't start anything

yvanzo: around?

zas: is this repo our DB setup? https://github.com/metabrainz/docker-postgres

[listenbrainz-server] alastair opened pull request #1185 (master…jenkins-tests): Add run script for jenkins integration tests, fix jenkins unit tests https://github.com/metabrainz/listenbrainz-serv...

ruaok: likely, but image is postgres-floyd

k

no dockerfile

ping me if there's anything I can help with

ruaok: we can try to rebuild the container, but that's quite random

yvanzo, bitmap: any of you around?

I don't know enough about current pg setup, and I know bitmap did a lot of changes recently

I'm digging.

try restarting pg bouncer, zas

how?

docker container.

postgres floyd is up

rarg what is happen

ruaok: pgbouncer is in postgres-floyd container