9:00 AM
iliekcomputers says hi from Bangkok 😊
2018-01-04 00406, 2018
9:01 AM
ruaok waves at iliekcomputers
2018-01-04 00426, 2018
9:01 AM
ruaok
LB is down, but likely not our fault. the DB is freaking out. but we have a poor error message up there.
2018-01-04 00421, 2018
9:02 AM
iliekcomputers
Yikes.
2018-01-04 00446, 2018
9:02 AM
iliekcomputers
ruaok: open a ticket and I will take a look today?
2018-01-04 00404, 2018
9:03 AM
ruaok
ok.
2018-01-04 00421, 2018
9:03 AM
ruaok
first I gotta try and keep the sites up. everything is tipping over.
2018-01-04 00439, 2018
9:04 AM
iliekcomputers can't wait for Sentry to start working with LB tbh
2018-01-04 00405, 2018
9:10 AM
vishalchoudhary[ joined the channel
2018-01-04 00404, 2018
9:17 AM
djwhitey joined the channel
2018-01-04 00457, 2018
9:18 AM
djwhitey_ has quit
2018-01-04 00417, 2018
9:23 AM
xps2 has quit
2018-01-04 00450, 2018
9:27 AM
xps2 joined the channel
2018-01-04 00426, 2018
9:30 AM
samj1912
ruaok: sorry, just woke up
2018-01-04 00437, 2018
9:31 AM
ruaok
k. postgres is freaking out again.
2018-01-04 00450, 2018
9:31 AM
ruaok
I stopped sir & solr, no change (not surprising)
2018-01-04 00405, 2018
9:32 AM
ruaok
running a vacuum analyze now, no improvement so far.
2018-01-04 00414, 2018
9:32 AM
ruaok
do you remember what else helped in the past?
2018-01-04 00430, 2018
9:32 AM
samj1912
ruaok, vaccum analyze is what you want
2018-01-04 00437, 2018
9:32 AM
samj1912
It should work once it's done
2018-01-04 00406, 2018
9:33 AM
ruaok
do we not have the auto-vacuum thingy running?
2018-01-04 00416, 2018
9:35 AM
samj1912
PG does. I think the problem is once sir does a re index with recordings, it somehow messes up PG
2018-01-04 00440, 2018
9:35 AM
samj1912
In the past we have had such problems only when sir did a complete reindex
2018-01-04 00457, 2018
9:35 AM
ruaok
and we just did, no?
2018-01-04 00402, 2018
9:36 AM
samj1912
Yup
2018-01-04 00414, 2018
9:36 AM
ruaok
from now on, when you re-index. run a vacuum anayze!
2018-01-04 00423, 2018
9:36 AM
ruaok gives samj1912 a light spanking
2018-01-04 00423, 2018
9:36 AM
samj1912
But this time I did it against Master (bowie)
2018-01-04 00444, 2018
9:36 AM
samj1912
Last time it was against queen
2018-01-04 00445, 2018
9:36 AM
ruaok
clearly it matters not.
2018-01-04 00404, 2018
9:37 AM
samj1912
We thought it was hot standby feedback that was causing the problem
2018-01-04 00416, 2018
9:37 AM
samj1912
From today, it seems not
2018-01-04 00429, 2018
9:37 AM
samj1912
Because PG should technically be doing this on its own.
2018-01-04 00435, 2018
9:37 AM
samj1912
Weird that it's not
2018-01-04 00454, 2018
9:37 AM
samj1912
I'll look into it with bitmap
2018-01-04 00439, 2018
9:39 AM
yvanzo
Is it possible to read docker logs from another place than bowie?
2018-01-04 00450, 2018
9:41 AM
yvanzo
docker logs --since takes forever
2018-01-04 00436, 2018
9:43 AM
samj1912
yvanzo: redirect to stdout and then do a head?
2018-01-04 00453, 2018
9:43 AM
samj1912
2018-01-04 00430, 2018
9:44 AM
samj1912
bitmap should be better informed about the value though
2018-01-04 00443, 2018
9:45 AM
yvanzo
samj1912: I used --tail
2018-01-04 00427, 2018
9:46 AM
yvanzo
(from the beginning)
2018-01-04 00400, 2018
9:47 AM
samj1912
Worked?
2018-01-04 00414, 2018
9:48 AM
yvanzo
Yes and no, it took several minutes and returned logs with unmatched timestamps.
2018-01-04 00410, 2018
9:49 AM
samj1912
I generally do docker logs --since xyz 2&>1 | head -n 50
2018-01-04 00430, 2018
9:49 AM
Anirudh joined the channel
2018-01-04 00434, 2018
9:49 AM
xps2 has quit
2018-01-04 00459, 2018
9:49 AM
Anirudh
Okay, so hey!
2018-01-04 00400, 2018
9:50 AM
yvanzo
Ok, sorry, it is tail, not head :/
2018-01-04 00413, 2018
9:50 AM
Anirudh
I'm Anirudh, and I'm participating in GCI
2018-01-04 00442, 2018
9:50 AM
Anirudh
And I don't know what's going on, would someone just update me??
2018-01-04 00456, 2018
9:50 AM
yvanzo
Welcome! Do you have any question?
2018-01-04 00417, 2018
9:51 AM
Anirudh
No, I just was messing around when I landed here.... So....
2018-01-04 00459, 2018
9:51 AM
Anirudh
I was actually thinking which task I should take up next
2018-01-04 00400, 2018
9:53 AM
samj1912
ruaok: `1 S lxd 20934 20929 0 80 0 - 8642073 - 2017 ? 00:01:28 postgres: autovacuum launcher process ` autovaccum is on and running since 2017
2018-01-04 00423, 2018
9:53 AM
samj1912
we will have to investigate why its not doing its job :\
2018-01-04 00430, 2018
9:53 AM
Anirudh has left the channel
2018-01-04 00453, 2018
9:56 AM
samj1912
ugh dammnit, `hot_standby_feedback` is on on bowie :\
2018-01-04 00457, 2018
9:56 AM
Vini joined the channel
2018-01-04 00408, 2018
9:57 AM
samj1912
I thought bitmap turned it off
2018-01-04 00423, 2018
9:57 AM
Major_Lurker
Hi guys I take it the 502 errors are related to the above?
2018-01-04 00441, 2018
9:57 AM
samj1912
Major_Lurker: yes
2018-01-04 00453, 2018
9:57 AM
Major_Lurker
thought so thanks.... good luck :)
2018-01-04 00405, 2018
9:58 AM
rahul_india joined the channel
2018-01-04 00426, 2018
9:58 AM
samj1912
its on bowie so it shouldn't matter but still
2018-01-04 00433, 2018
9:58 AM
samj1912
ill ask bitmap to remove it
2018-01-04 00454, 2018
9:59 AM
rahul_india
2018-01-04 00407, 2018
10:00 AM
rahul_india
server*
2018-01-04 00431, 2018
10:00 AM
Major_Lurker
rahul_india, just wait they are working on it
2018-01-04 00452, 2018
10:00 AM
rahul_india
okay
2018-01-04 00430, 2018
10:01 AM
rahul_india has quit
2018-01-04 00421, 2018
10:06 AM
samj1912
ruaok: is the vaccum still running?
2018-01-04 00453, 2018
10:06 AM
samj1912
*vacuum
2018-01-04 00430, 2018
10:17 AM
ruaok
it finished.
2018-01-04 00442, 2018
10:17 AM
ruaok
sigh
2018-01-04 00459, 2018
10:17 AM
samj1912
just now?
2018-01-04 00418, 2018
10:18 AM
ruaok
dunno. I went back to napping, since I've got the flu
2018-01-04 00442, 2018
10:18 AM
samj1912
I dont think it helped
2018-01-04 00416, 2018
10:19 AM
ruaok
sure doesn't look that way.
2018-01-04 00436, 2018
10:19 AM
ruaok
but, I have no ideas on what else it could be.
2018-01-04 00440, 2018
10:19 AM
ruaok
I can't think straight.
2018-01-04 00405, 2018
10:20 AM
samj1912
should we restart postgres?
2018-01-04 00439, 2018
10:20 AM
Vini has quit
2018-01-04 00408, 2018
10:21 AM
ruaok
2018-01-04 00427, 2018
10:21 AM
ruaok
do you know how to do that?
2018-01-04 00432, 2018
10:21 AM
samj1912
nope
2018-01-04 00411, 2018
10:24 AM
samj1912
just docker restart wont do?
2018-01-04 00457, 2018
10:27 AM
ruaok
I dont know. nor do we want to corrupt the DB
2018-01-04 00437, 2018
10:28 AM
samj1912
ruaok: did you run vacuum as postgres or musicbrainz?
2018-01-04 00448, 2018
10:28 AM
ruaok
musicbrainz
2018-01-04 00459, 2018
10:28 AM
samj1912
running it again seems to timeout
2018-01-04 00421, 2018
10:29 AM
samj1912
2018-01-04 00440, 2018
10:29 AM
samj1912
should I redo it with set_statement_timeout 0?
2018-01-04 00443, 2018
10:29 AM
samj1912
yeah
2018-01-04 00459, 2018
10:29 AM
ruaok
it won't help to do it again.
2018-01-04 00430, 2018
10:30 AM
ruaok
if it didn't fix anything, running again won't improve it.
2018-01-04 00425, 2018
10:31 AM
samj1912
postgres doesn't have access to musicbrainz_db right?
2018-01-04 00444, 2018
10:31 AM
ruaok
it should, why?
2018-01-04 00444, 2018
10:35 AM
ruaok
we could try re-starting all the -web and -ws containers
2018-01-04 00448, 2018
10:35 AM
ruaok
see if that improves things.
2018-01-04 00457, 2018
10:35 AM
samj1912
okay
2018-01-04 00450, 2018
10:38 AM
yvanzo
Can we stop sentry btw?
2018-01-04 00439, 2018
10:39 AM
yvanzo
It is ~10% of queries.
2018-01-04 00424, 2018
10:40 AM
ruaok
sure, we've done that in the past.
2018-01-04 00430, 2018
10:40 AM
rvedotrc has quit
2018-01-04 00415, 2018
10:43 AM
Vini joined the channel
2018-01-04 00404, 2018
10:45 AM
xps2 joined the channel
2018-01-04 00434, 2018
10:45 AM
rvedotrc joined the channel
2018-01-04 00451, 2018
10:45 AM
yvanzo
done
2018-01-04 00403, 2018
10:49 AM
samj1912
2018-01-04 00414, 2018
10:49 AM
samj1912
still running
2018-01-04 00421, 2018
10:53 AM
samj1912
last time I think it took 2 hrs when bitmap ran it
2018-01-04 00414, 2018
10:56 AM
Dragonzeron has quit
2018-01-04 00424, 2018
10:57 AM
Vini has quit
2018-01-04 00448, 2018
11:02 AM
samj1912
seems to be getting better?
2018-01-04 00400, 2018
11:03 AM
ruaok
no
2018-01-04 00430, 2018
11:03 AM
ruaok
maybe.
2018-01-04 00432, 2018
11:03 AM
samj1912
ouch, load back to 40 :\
2018-01-04 00439, 2018
11:03 AM
ruaok
the CPU on bowie is the key to look for.
2018-01-04 00450, 2018
11:03 AM
ruaok
it dropped for a brief moment, but back above 90%
2018-01-04 00425, 2018
11:04 AM
ruaok
we need to think about other things. vacuum analyze didn't work, nor will it work the second time.
2018-01-04 00455, 2018
11:04 AM
samj1912
its not from the second time? given the age its from the first time?
2018-01-04 00423, 2018
11:05 AM
ruaok
I ran one on bowie.
2018-01-04 00406, 2018
11:06 AM
samj1912
yeah, it seems to be that one?
2018-01-04 00432, 2018
11:07 AM
samj1912
I wasn't up at the time it ran
2018-01-04 00405, 2018
11:08 AM
ruaok
your last two statements make no sense to me
2018-01-04 00411, 2018
11:08 AM
ruaok
oh
2018-01-04 00419, 2018
11:08 AM
ruaok
nm.
2018-01-04 00436, 2018
11:08 AM
ruaok
you weren't up... got it.
2018-01-04 00400, 2018
11:10 AM
samj1912
meanwhile, what should we do, I restarted a couple of containers
2018-01-04 00405, 2018
11:10 AM
samj1912
didn't seem to help
2018-01-04 00452, 2018
11:11 AM
ruaok
restarting postgres is probably the right thing to do, but without knowing how...
2018-01-04 00419, 2018
11:12 AM
samj1912
yvanzo: do you have any idea?
2018-01-04 00459, 2018
11:20 AM
yvanzo
samj1912: turning hot_standby_feedback off again?
2018-01-04 00413, 2018
11:21 AM
samj1912
yvanzo: we will have to restart pg for that
2018-01-04 00425, 2018
11:21 AM
samj1912
and its just `on` on the master
2018-01-04 00433, 2018
11:21 AM
samj1912
pg docs say it shouldn't matter there
2018-01-04 00426, 2018
11:22 AM
samj1912
I am just waiting for the vacuum analyze to complete, its almost running since 2 hrs now, should be near completion
2018-01-04 00437, 2018
11:23 AM
yvanzo
samj1912: hot_standby is 'on' on bowie/master, and hot_standby_feedback is 'on' on queen/slave, that would just require restarting pg slave. But I'm not sure it would matter really.
2018-01-04 00409, 2018
11:24 AM
samj1912
hot_standby_feedback should be off on queen? O.o
2018-01-04 00426, 2018
11:24 AM
samj1912
im pretty sure bitmap turned it off
2018-01-04 00459, 2018
11:24 AM
samj1912
yup, its off
2018-01-04 00426, 2018
11:25 AM
samj1912
hot_standby is `on` on queen
2018-01-04 00449, 2018
11:25 AM
yvanzo
Oops indeed, it is the opposite
2018-01-04 00458, 2018
11:25 AM
yvanzo
:(
2018-01-04 00435, 2018
11:26 AM
samj1912
yvanzo: do you know how to restart pg master?
2018-01-04 00451, 2018
11:29 AM
Vini joined the channel
2018-01-04 00412, 2018
11:30 AM
yvanzo
nope
2018-01-04 00446, 2018
11:39 AM
naiveai
Leo_Verto: do I base my branch off of pipenv-docker?
2018-01-04 00448, 2018
11:42 AM
samj1912
yvanzo: do you have any idea what musicbrainz_ro does?
