#metabrainz

LB is down, but likely not our fault. the DB is freaking out. but we have a poor error message up there.

Yikes.

ruaok: open a ticket and I will take a look today?

ok.

first I gotta try and keep the sites up. everything is tipping over.

ruaok: sorry, just woke up

k. postgres is freaking out again.

I stopped sir & solr, no change (not surprising)

running a vacuum analyze now, no improvement so far.

do you remember what else helped in the past?

ruaok, vaccum analyze is what you want

It should work once it's done

do we not have the auto-vacuum thingy running?

PG does. I think the problem is once sir does a re index with recordings, it somehow messes up PG

In the past we have had such problems only when sir did a complete reindex

and we just did, no?

Yup

from now on, when you re-index. run a vacuum anayze!

But this time I did it against Master (bowie)

Last time it was against queen

clearly it matters not.

We thought it was hot standby feedback that was causing the problem

From today, it seems not

Because PG should technically be doing this on its own.

Weird that it's not

I'll look into it with bitmap

Is it possible to read docker logs from another place than bowie?

docker logs --since takes forever

yvanzo: redirect to stdout and then do a head?

ruaok: https://stackoverflow.com/questions/19527571/po...

bitmap should be better informed about the value though

samj1912: I used --tail

(from the beginning)

Worked?

Yes and no, it took several minutes and returned logs with unmatched timestamps.

I generally do docker logs --since xyz 2&>1 | head -n 50

Okay, so hey!

Ok, sorry, it is tail, not head :/

I'm Anirudh, and I'm participating in GCI

And I don't know what's going on, would someone just update me??

Welcome! Do you have any question?

No, I just was messing around when I landed here.... So....

I was actually thinking which task I should take up next

ruaok: `1 S lxd 20934 20929 0 80 0 - 8642073 - 2017 ? 00:01:28 postgres: autovacuum launcher process ` autovaccum is on and running since 2017

we will have to investigate why its not doing its job :\

ugh dammnit, `hot_standby_feedback` is on on bowie :\

I thought bitmap turned it off

Hi guys I take it the 502 errors are related to the above?

thought so thanks.... good luck :)

its on bowie so it shouldn't matter but still

ill ask bitmap to remove it

hello, I'm unable to access https://musicbrainz.org/ its showing error 502 "serwer is being naughty". what should i do?

server*

rahul_india, just wait they are working on it

okay

ruaok: is the vaccum still running?

*vacuum

it finished.

sigh

just now?

dunno. I went back to napping, since I've got the flu

I dont think it helped

sure doesn't look that way.

but, I have no ideas on what else it could be.

I can't think straight.

should we restart postgres?

zas doesn't either and he is: https://www.flightradar24.com/EZY75LF/1006f125

do you know how to do that?

nope

just docker restart wont do?

I dont know. nor do we want to corrupt the DB

ruaok: did you run vacuum as postgres or musicbrainz?

musicbrainz

running it again seems to timeout

should I redo it with set_statement_timeout 0?

yeah

it won't help to do it again.

if it didn't fix anything, running again won't improve it.

postgres doesn't have access to musicbrainz_db right?

it should, why?

we could try re-starting all the -web and -ws containers

see if that improves things.

okay

Can we stop sentry btw?

It is ~10% of queries.

sure, we've done that in the past.

done

still running

last time I think it took 2 hrs when bitmap ran it

seems to be getting better?

no

maybe.

ouch, load back to 40 :\

the CPU on bowie is the key to look for.

it dropped for a brief moment, but back above 90%

we need to think about other things. vacuum analyze didn't work, nor will it work the second time.

its not from the second time? given the age its from the first time?

I ran one on bowie.

yeah, it seems to be that one?

I wasn't up at the time it ran

your last two statements make no sense to me

oh

nm.

you weren't up... got it.

meanwhile, what should we do, I restarted a couple of containers

didn't seem to help

restarting postgres is probably the right thing to do, but without knowing how...

yvanzo: do you have any idea?

samj1912: turning hot_standby_feedback off again?

yvanzo: we will have to restart pg for that

and its just `on` on the master

pg docs say it shouldn't matter there

I am just waiting for the vacuum analyze to complete, its almost running since 2 hrs now, should be near completion

samj1912: hot_standby is 'on' on bowie/master, and hot_standby_feedback is 'on' on queen/slave, that would just require restarting pg slave. But I'm not sure it would matter really.

hot_standby_feedback should be off on queen? O.o

im pretty sure bitmap turned it off

yup, its off

hot_standby is `on` on queen

Oops indeed, it is the opposite

:(

yvanzo: do you know how to restart pg master?

nope

Leo_Verto: do I base my branch off of pipenv-docker?

yvanzo: do you have any idea what musicbrainz_ro does?