19:19 PM
zas
ruaok: nc to ernie is prolly block by firewall rules
2016-05-23 14444, 2016
19:19 PM
zas
blocked*
2016-05-23 14454, 2016
19:19 PM
ruaok
yeah, likely.
2016-05-23 14402, 2016
19:20 PM
ruaok
should've figured that
2016-05-23 14410, 2016
19:20 PM
chirlu
hellska: No, no meeting. (And no need to [off] the question it. :) )
2016-05-23 14415, 2016
19:20 PM
chirlu
-it
2016-05-23 14436, 2016
19:20 PM
hellska
Ok, thanx :)
2016-05-23 14407, 2016
19:23 PM
Gentlecat
hellska: just keep coding :P
2016-05-23 14427, 2016
19:23 PM
Gentlecat
I think the coding period officially started today
2016-05-23 14412, 2016
19:24 PM
hellska
gentlecat: yeah what do you think I’m doing ;)
2016-05-23 14433, 2016
19:24 PM
Gentlecat
👍
2016-05-23 14439, 2016
19:24 PM
hellska
and definitely yes it started today!
2016-05-23 14413, 2016
19:25 PM
zas
bitmap, ruaok : baron looks to work, load is "normal", website is a bit slower
2016-05-23 14413, 2016
19:25 PM
ruaok
zas: ready for pg95 backup?
2016-05-23 14419, 2016
19:25 PM
aron_kexp joined the channel
2016-05-23 14422, 2016
19:25 PM
bitmap
good
2016-05-23 14424, 2016
19:25 PM
ruaok
yeah, decent.
2016-05-23 14430, 2016
19:25 PM
ruaok
zas: go!
2016-05-23 14449, 2016
19:26 PM
zas
in progress
2016-05-23 14454, 2016
19:26 PM
ruaok
th
2016-05-23 14455, 2016
19:26 PM
ruaok
x
2016-05-23 14410, 2016
19:29 PM
chirlu
Load on baron is rising.
2016-05-23 14442, 2016
19:29 PM
chirlu
Since it has fewer cores, the number of backends to be used by pgbouncer on it should perhaps be reduced accordingly?
2016-05-23 14438, 2016
19:30 PM
zas
it can do it i think, especially since it has to warm up caches
2016-05-23 14423, 2016
19:31 PM
ruaok
load is less than 2 right now.
2016-05-23 14408, 2016
19:37 PM
chirlu disappears for half an hour.
2016-05-23 14442, 2016
19:43 PM
JonnyJD joined the channel
2016-05-23 14426, 2016
19:49 PM
mackie has quit
2016-05-23 14400, 2016
19:50 PM
bitmap
I'm gonna stop cron on pino to shut up gamekeeper
2016-05-23 14425, 2016
19:50 PM
FergusL
the query webservice is down again? (I'm getting rate limit error)
2016-05-23 14457, 2016
19:50 PM
zas
FergusL: huge slow down atm, we are transferring huge data, and it has bad impact on services
2016-05-23 14406, 2016
19:51 PM
zas
should be better in 10 mins
2016-05-23 14413, 2016
19:51 PM
FergusL
ok ok, I better just wait
2016-05-23 14425, 2016
19:51 PM
FergusL
good luck with all this team :)
2016-05-23 14432, 2016
19:51 PM
zas
expect down time later, we are moving to new schema
2016-05-23 14403, 2016
19:57 PM
hellska has quit
2016-05-23 14426, 2016
19:57 PM
hellska joined the channel
2016-05-23 14408, 2016
19:59 PM
ruaok
DONE
2016-05-23 14449, 2016
19:59 PM
Freso
WOO!
2016-05-23 14406, 2016
20:03 PM
bitmap
starting ./upgrade.sh
2016-05-23 14429, 2016
20:04 PM
zas sacrifices the last black chicken, mixing blood and rum, ...
2016-05-23 14400, 2016
20:05 PM
Gentlecat
zas: add some gelato
2016-05-23 14404, 2016
20:05 PM
bitmap
running
2016-05-23 14404, 2016
20:05 PM
ruaok
running upgrade.sh
2016-05-23 14441, 2016
20:05 PM
zas looks for the infamous voodoo gelato ...
2016-05-23 14443, 2016
20:06 PM
ruaok
zas: the site isn't working because pingu and maybe another is throwing the same cant connect to socket error from yesterday.
2016-05-23 14456, 2016
20:06 PM
ruaok
can you please restart both back ends on pingu?
2016-05-23 14459, 2016
20:06 PM
zas
hmmm interesting
2016-05-23 14459, 2016
20:07 PM
ruaok
asterix is fune
2016-05-23 14403, 2016
20:08 PM
ruaok
fine too
2016-05-23 14428, 2016
20:08 PM
zas
done on pingu
2016-05-23 14438, 2016
20:08 PM
zas
really strange issue
2016-05-23 14409, 2016
20:09 PM
zas
i will restart bert nginx too
2016-05-23 14458, 2016
20:09 PM
zas
hmmm i think about something
2016-05-23 14410, 2016
20:10 PM
zas
errors are related to unix sockets right ?
2016-05-23 14421, 2016
20:10 PM
ruaok
yes
2016-05-23 14429, 2016
20:10 PM
ruaok
seem local to the web server though
2016-05-23 14431, 2016
20:10 PM
zas
and when backends are slowing down number of open sockets increase
2016-05-23 14441, 2016
20:10 PM
ruaok
yup, make sense
2016-05-23 14444, 2016
20:10 PM
zas
perhaps just no enough sockets
2016-05-23 14447, 2016
20:10 PM
zas
not*
2016-05-23 14457, 2016
20:10 PM
ruaok
load on baron is improving
2016-05-23 14413, 2016
20:11 PM
ruaok
possibly
2016-05-23 14436, 2016
20:12 PM
chirlu is back.
2016-05-23 14450, 2016
20:12 PM
chirlu
Just in case anybody cares. ;-)
2016-05-23 14409, 2016
20:13 PM
ruaok
my life is always harder when you're not around. I care. :)
2016-05-23 14443, 2016
20:13 PM
aron_kexp has quit
2016-05-23 14428, 2016
20:14 PM
bitmap
2016-05-23 14420, 2016
20:17 PM
FergusL
2016-05-23 14444, 2016
20:17 PM
chirlu
That’s external, shouldn’t be affected.
2016-05-23 14400, 2016
20:18 PM
ariscop has quit
2016-05-23 14403, 2016
20:19 PM
FergusL
thanks
2016-05-23 14409, 2016
20:21 PM
bitmap
2016-05-23 14449, 2016
20:22 PM
bitmap
upgrade.sh aborted due to critical human error
2016-05-23 14421, 2016
20:26 PM
Freso
Pesky humans.
2016-05-23 14429, 2016
20:30 PM
bitmap
2016-05-23 14435, 2016
20:30 PM
bitmap
"No packets available for this period"
2016-05-23 14459, 2016
20:33 PM
bitmap
upgrade.sh running minus BundleReplicationPackets
2016-05-23 14415, 2016
20:38 PM
CatQuest
jesus christ you guys are still going at it 24 hours later?
2016-05-23 14416, 2016
20:38 PM
CatQuest
you guys deserve some norwegian chocolate!
2016-05-23 14416, 2016
20:38 PM
CatQuest
I'll send some to the office, ok?
2016-05-23 14440, 2016
20:38 PM
Zastai has quit
2016-05-23 14413, 2016
20:39 PM
bitmap
cancelled again so we can fix that
2016-05-23 14440, 2016
20:39 PM
CatQuest goes to bed
2016-05-23 14441, 2016
20:39 PM
CatQuest
good luch you guys. fingers and toes and thumbs crossed
2016-05-23 14402, 2016
20:40 PM
CatQuest
luck even.
2016-05-23 14403, 2016
20:40 PM
CatQuest
let that not be an omen
2016-05-23 14416, 2016
20:40 PM
ruaok
CatQuest: ok!
2016-05-23 14421, 2016
20:40 PM
CatQuest
ヽ༼ຈل͜ຈ༽ﾉ
2016-05-23 14402, 2016
20:41 PM
CatQuest
ruaok: any spesific requests?
2016-05-23 14421, 2016
20:41 PM
ruaok
milk chocolate. :)
2016-05-23 14441, 2016
20:41 PM
CatQuest doesnt usually send or bring non-milk
2016-05-23 14419, 2016
20:42 PM
CatQuest
aha! i know, ill send a specia lsuprise for freso. even thoguh it will arrive before him :D
2016-05-23 14402, 2016
20:43 PM
CatQuest
:yawn: good luck you guys. here's to it working smootly now
2016-05-23 14430, 2016
20:43 PM
bitmap
update.sh running, this time bundling packets is working
2016-05-23 14434, 2016
20:43 PM
yeeeargh has quit
2016-05-23 14437, 2016
20:43 PM
bitmap
upgrade
2016-05-23 14401, 2016
20:52 PM
ruaok
am I reading this right, but we're handling 450 reqs/s with baron?
2016-05-23 14404, 2016
20:52 PM
ariscop joined the channel
2016-05-23 14446, 2016
20:52 PM
zas
yup
2016-05-23 14428, 2016
20:53 PM
ruaok pats zas on the back
2016-05-23 14429, 2016
20:53 PM
michiwend has quit
2016-05-23 14430, 2016
20:53 PM
ruaok
!m zas
2016-05-23 14430, 2016
20:53 PM
BrainzBot
You're doing good work, zas!
2016-05-23 14434, 2016
20:53 PM
zas
;)
2016-05-23 14419, 2016
20:55 PM
michiwend joined the channel
2016-05-23 14454, 2016
20:55 PM
zas
but the load on web servers (4) atm is a bit high, so we can't go much higher
2016-05-23 14436, 2016
20:57 PM
regagain has quit
2016-05-23 14400, 2016
20:58 PM
ruaok
yeah that is because we took astro out for the schema change
2016-05-23 14459, 2016
21:07 PM
zas
bitmap: do you need help on something ?
2016-05-23 14410, 2016
21:08 PM
mihaitish has quit
2016-05-23 14458, 2016
21:08 PM
ruaok
everyone is in the kitchen eating
2016-05-23 14457, 2016
21:09 PM
zas
lol, can't help ;)
2016-05-23 14429, 2016
21:16 PM
chirlu is experi-mint-ing, with a peppermint couscous.
2016-05-23 14413, 2016
21:19 PM
ruaok
groan'
2016-05-23 14405, 2016
21:24 PM
Clint twitches.
2016-05-23 14435, 2016
21:29 PM
Leftmost applauds.
2016-05-23 14447, 2016
21:29 PM
Freso <slowclap.gif>
2016-05-23 14407, 2016
21:33 PM
bitmap
tail -f /home/musicbrainz/musicbrainz-server/upgrade.log
2016-05-23 14414, 2016
21:33 PM
ruaok
he did screen it
2016-05-23 14423, 2016
21:33 PM
ruaok
geh, wrong window.
2016-05-23 14455, 2016
21:42 PM
CallerNo6
when chatlogs are back up, I want to do a search for all the things people say right before "wrong [tab|window]" . surely there's something to be learned.
2016-05-23 14429, 2016
21:46 PM
chirlu
Chatlogs are up, aren’t they?
2016-05-23 14446, 2016
21:46 PM
MajorLurker joined the channel
2016-05-23 14435, 2016
21:48 PM
chirlu
So, how far has the upgrade script come=
2016-05-23 14431, 2016
21:49 PM
bitmap
2016-05-23 14451, 2016
21:50 PM
bitmap
it was running before and mysteriously failed somewhere, though
2016-05-23 14431, 2016
21:52 PM
chirlu
Quite far then, actually. I’d expect only the reclustering to take a longer time after that.
2016-05-23 14405, 2016
21:55 PM
CallerNo6
chirlu, I get "Firefox can't establish a connection to the server at chatlogs.metabrainz.org"
2016-05-23 14422, 2016
21:56 PM
chirlu
It seems https is down for some reason. Unencrypted is working. zas?
2016-05-23 14439, 2016
21:56 PM
Freso
CallerNo6: Known issue.
2016-05-23 14402, 2016
21:57 PM
zas
well, it does... but hsts shit will cause your browser to be redirected to https which doesnt work yet
2016-05-23 14402, 2016
21:57 PM
Freso
CallerNo6: Because metabrainz.org's HSTS policy is set to also be for subdomains.
2016-05-23 14414, 2016
21:57 PM
zas
i removed it, but it is cached
2016-05-23 14409, 2016
21:58 PM
Freso
chirlu: HTTPS for chatlogs never worked, so it was never "up".
2016-05-23 14431, 2016
21:58 PM
Freso
LordSputnik is on it and they will be available over HTTPS within a few weeks.
2016-05-23 14437, 2016
22:00 PM
CallerNo6
right. I get a 200 hundred from http: but a "connection refused" from https:
2016-05-23 14459, 2016
22:00 PM
Freso
2016-05-23 14415, 2016
22:01 PM
Freso
(Or possibly clear browser cache?)
2016-05-23 14430, 2016
22:02 PM
zas
2016-05-23 14432, 2016
22:02 PM
bitmap
INSERT 0 35283872
2016-05-23 14432, 2016
22:03 PM
zas
i disabled hsts for metabrainz (max-age=0) so browsers will uncache hsts info for chatlogs
2016-05-23 14457, 2016
22:03 PM
zas
tell me if it works for you
2016-05-23 14421, 2016
22:05 PM
zas
chatlogs isn't on https yet, planned though
2016-05-23 14405, 2016
22:09 PM
CallerNo6
cleared cache (whether I needed to or not). all is well.
2016-05-23 14452, 2016
22:10 PM
bitmap
on to 20160515-recluster.sql
2016-05-23 14403, 2016
22:11 PM
chirlu has quit
2016-05-23 14447, 2016
22:11 PM
chirlu joined the channel
2016-05-23 14434, 2016
22:16 PM
Gentlecat is back home
2016-05-23 14406, 2016
22:18 PM
bitmap
wb
