0:00 AM
MusicbrainzB0T2 joined the channel
2018-07-02 18326, 2018
0:04 AM
MusicbrainzB0T has quit
2018-07-02 18350, 2018
1:01 AM
Nyanko-sensei joined the channel
2018-07-02 18327, 2018
1:03 AM
D4RK-PH0ENiX has quit
2018-07-02 18324, 2018
1:12 AM
Nyanko-sensei has quit
2018-07-02 18357, 2018
1:12 AM
D4RK-PH0ENiX joined the channel
2018-07-02 18302, 2018
1:58 AM
samj1912
onculayn: weird, which picard version is that?
2018-07-02 18317, 2018
1:58 AM
samj1912
Are you using picard-dev from pypi?
2018-07-02 18307, 2018
2:13 AM
Lotheric has quit
2018-07-02 18357, 2018
2:13 AM
Lotheric joined the channel
2018-07-02 18318, 2018
2:23 AM
christomo joined the channel
2018-07-02 18327, 2018
2:56 AM
bukwurm joined the channel
2018-07-02 18317, 2018
3:12 AM
Leo__Verto has quit
2018-07-02 18342, 2018
3:16 AM
Leo__Verto joined the channel
2018-07-02 18346, 2018
3:21 AM
Leo__Verto has quit
2018-07-02 18325, 2018
3:29 AM
thomasross joined the channel
2018-07-02 18351, 2018
4:04 AM
samj1912
Oh btw which python3 version are you using?
2018-07-02 18358, 2018
4:04 AM
samj1912
Picard requires 3.5 minimum
2018-07-02 18336, 2018
4:43 AM
suhas2go joined the channel
2018-07-02 18302, 2018
5:54 AM
UmkaDK_ joined the channel
2018-07-02 18337, 2018
5:55 AM
UmkaDK has quit
2018-07-02 18307, 2018
7:10 AM
christomo has quit
2018-07-02 18314, 2018
7:18 AM
samj1912
zas, you up?
2018-07-02 18311, 2018
8:06 AM
zas
yes
2018-07-02 18350, 2018
8:12 AM
samj1912
Wanna try setting up a replica?
2018-07-02 18358, 2018
8:12 AM
samj1912
I wanna see how long it takes
2018-07-02 18340, 2018
8:15 AM
UmkaDK joined the channel
2018-07-02 18355, 2018
8:16 AM
UmkaDK_ has quit
2018-07-02 18315, 2018
8:22 AM
UmkaDK has quit
2018-07-02 18358, 2018
8:23 AM
UmkaDK joined the channel
2018-07-02 18335, 2018
8:35 AM
reosarevok
Hmm, I'm getting a lot of "an error ocurred while searching" in inline search
2018-07-02 18341, 2018
8:35 AM
reosarevok
(that work perfectly fine at the next attempt)
2018-07-02 18303, 2018
8:37 AM
reosarevok
Any idea what's going on?
2018-07-02 18311, 2018
8:37 AM
reosarevok
samj1912, zas ^^
2018-07-02 18343, 2018
8:37 AM
zas
hmmm let me see, i made a change that could cause that
2018-07-02 18350, 2018
8:37 AM
zas
which request ?
2018-07-02 18315, 2018
8:38 AM
ruaok
what change did you make?
2018-07-02 18333, 2018
8:38 AM
reosarevok
zas: everything, at seemingly random, in inline search
2018-07-02 18342, 2018
8:39 AM
zas
just with inline search?
2018-07-02 18356, 2018
8:39 AM
zas
tried few queries and all worked
2018-07-02 18348, 2018
8:40 AM
zas
hmmm i see there's an increase in 4xx
2018-07-02 18314, 2018
8:46 AM
reosarevok
No, I got something with normal now, "not enough info" or something, retry worked
2018-07-02 18335, 2018
8:46 AM
reosarevok
2018-07-02 18331, 2018
8:56 AM
reosarevok
zas: still getting a ton of them
2018-07-02 18334, 2018
8:56 AM
reosarevok
In beta, if it matters
2018-07-02 18355, 2018
8:56 AM
zas
i don't see which changes i made could cause that
2018-07-02 18327, 2018
8:57 AM
zas
i had to reload haproxy config to change backend weights, but i didn't change anything else
2018-07-02 18334, 2018
8:57 AM
zas
samj1912: ^^ any idea?
2018-07-02 18321, 2018
9:02 AM
zas
i switched haproxy to lb2 to see if it happens on both nodes, if it does i suspect a solr issue
2018-07-02 18336, 2018
9:03 AM
zas
i upgraded, rebooted, and switched back to lb1
2018-07-02 18342, 2018
9:03 AM
reosarevok
Maybe one server is having issues?
2018-07-02 18351, 2018
9:03 AM
reosarevok
It's weird it works instantly sometimes, and not at all others
2018-07-02 18356, 2018
9:03 AM
reosarevok
(also instantly)
2018-07-02 18307, 2018
9:04 AM
samj1912
Huh
2018-07-02 18312, 2018
9:04 AM
samj1912
Weird
2018-07-02 18339, 2018
9:04 AM
samj1912
Probably related to one of the solr servers then
2018-07-02 18313, 2018
9:10 AM
samj1912
Huh weird, it's sometimes returning results and sometimes not
2018-07-02 18304, 2018
9:19 AM
samj1912
zas, can't find anything wrong but 4xx suddenly increased after 14:00 IST
2018-07-02 18352, 2018
9:29 AM
samj1912
zas, did you change something recently?
2018-07-02 18301, 2018
9:30 AM
samj1912
Solr response times went to shit
2018-07-02 18326, 2018
9:32 AM
samj1912
zas, I am restarting all solr servers
2018-07-02 18331, 2018
9:32 AM
samj1912
Something weird is happening
2018-07-02 18344, 2018
9:32 AM
samj1912
Lots of 4xx suddenly
2018-07-02 18331, 2018
9:34 AM
ruaok
yvanzo: FWIW the -vm build showed zero errors -- worked as expected.
2018-07-02 18358, 2018
9:35 AM
alastairp
2018-07-02 18312, 2018
9:37 AM
samj1912
also zas can you break those response codes by solr nodes?
2018-07-02 18315, 2018
9:37 AM
ruaok
interesting. what happened circa 19h and 4:30h?
2018-07-02 18324, 2018
9:37 AM
alastairp
oh, I added more ice :)
2018-07-02 18314, 2018
9:38 AM
ruaok
do you have an icebox and then you ventilate from it?
2018-07-02 18335, 2018
9:38 AM
reosarevok
Does the icebox have plums?
2018-07-02 18339, 2018
9:38 AM
alastairp
yes, there's a fan that pulls cold air from an ice compartment into the main compartment
2018-07-02 18343, 2018
9:38 AM
alastairp
reosarevok: they're delicious
2018-07-02 18310, 2018
9:39 AM
reosarevok is reminded of https://twitter.com/hels/status/1012449490396237824
2018-07-02 18323, 2018
9:39 AM
alastairp
(I also lowered the target temperature at 6pm and 4am, so it pulled more cold air in)
2018-07-02 18327, 2018
9:40 AM
ZaphodBeeblebrox is now known as CatQuest
2018-07-02 18343, 2018
9:40 AM
alastairp
with enough ice I can definitely lower the temperature 10° lower than ambient. I've not tried to pull it down more
2018-07-02 18356, 2018
9:40 AM
ruaok
nice.
2018-07-02 18304, 2018
9:47 AM
reosarevok
No, ice
2018-07-02 18302, 2018
9:50 AM
samj1912
zas: lots of logs like
2018-07-02 18306, 2018
9:50 AM
samj1912
2018-07-02 18321, 2018
9:50 AM
ruaok
proxy issue?
2018-07-02 18331, 2018
9:50 AM
yvanzo
ruaok: the most breaking change is that base image are now made for Ubuntu Cloud using cloud-init.
2018-07-02 18337, 2018
9:50 AM
samj1912
looks like it
2018-07-02 18352, 2018
9:50 AM
samj1912
2018-07-02 18357, 2018
9:50 AM
ruaok
yvanzo: ok, what is our next step then?
2018-07-02 18359, 2018
9:50 AM
samj1912
normal reqs should look like this
2018-07-02 18307, 2018
9:51 AM
yvanzo
VM network settings can probably be simplified too.
2018-07-02 18324, 2018
9:51 AM
ruaok
samj1912: I've pinged zas elsewhere too, hopefully we can get his attention.
2018-07-02 18344, 2018
9:51 AM
samj1912
cool
2018-07-02 18309, 2018
9:54 AM
yvanzo
ruaok: bento/ubuntu-16.04
2018-07-02 18334, 2018
9:54 AM
samj1912
reosarevok: the above is probably why all those no result responses are occuring
2018-07-02 18342, 2018
9:54 AM
reosarevok
Got it :)
2018-07-02 18358, 2018
9:54 AM
yvanzo
it is lighter and probably requires less customization
2018-07-02 18358, 2018
9:54 AM
reosarevok
I mean, they're instant, so it's not too bad yet
2018-07-02 18300, 2018
9:55 AM
iliekcomputers
2018-07-02 18339, 2018
9:55 AM
samj1912
XD
2018-07-02 18305, 2018
9:56 AM
iliekcomputers
😂
2018-07-02 18320, 2018
9:56 AM
zas
samj1912: i just changed to weights, and did a config reload, i see nothing on haproxy side which may cause this
2018-07-02 18346, 2018
9:56 AM
samj1912
zas: did you see the logs above?
2018-07-02 18350, 2018
9:57 AM
zas
lol i know, i deleted some lines in the whitelist.... unintentionnally
2018-07-02 18302, 2018
9:58 AM
ruaok
yvanzo: do you need me to make the changes?
2018-07-02 18333, 2018
9:58 AM
yvanzo
nope, I’m creating a branch for that
2018-07-02 18351, 2018
9:58 AM
samj1912
zas: one more thing btw - what happens when people with slave servers use solr?
2018-07-02 18353, 2018
9:58 AM
ruaok
iliekcomputers: that's funny since we're still working samj to recognize what a weekend is. :)
2018-07-02 18300, 2018
9:59 AM
samj1912
2018-07-02 18319, 2018
9:59 AM
samj1912
but that isnt currently possible with solr since non hetzner reqs are blocked
2018-07-02 18302, 2018
10:00 AM
ruaok
sounds like we need to setup another proxy for that.
2018-07-02 18323, 2018
10:02 AM
samj1912
okay 4xx are gone, but response times are still bad
2018-07-02 18358, 2018
10:04 AM
samj1912
2018-07-02 18352, 2018
10:05 AM
samj1912
msg rates got double in the last 12 hrs
2018-07-02 18307, 2018
10:06 AM
samj1912
was some new service deployed?
2018-07-02 18311, 2018
10:06 AM
zas
no idea, about both
2018-07-02 18340, 2018
10:06 AM
samj1912
ruaok: ^^ ?
2018-07-02 18338, 2018
10:08 AM
samj1912
2018-07-02 18316, 2018
10:09 AM
ruaok shrugs
2018-07-02 18317, 2018
10:09 AM
ruaok
no clue.
2018-07-02 18328, 2018
10:09 AM
ruaok
it doesn't look good, I can tell you that.
2018-07-02 18330, 2018
10:09 AM
ruaok
#nothelping
2018-07-02 18320, 2018
10:10 AM
samj1912
did anything change at around 11:03? (related to solr)
2018-07-02 18350, 2018
10:10 AM
ruaok
I think you're the only person tinkering with this.
2018-07-02 18314, 2018
10:11 AM
samj1912
nope, wasnt touching solr around that time
2018-07-02 18333, 2018
10:11 AM
samj1912
also those rmq things are due to BQ jobs
2018-07-02 18343, 2018
10:11 AM
samj1912
just checked the rmq admin dash
2018-07-02 18354, 2018
10:11 AM
ruaok
oh, yeah first of the month.
2018-07-02 18321, 2018
10:12 AM
ruaok
a bit late in the day though. iliekcomputers?
2018-07-02 18344, 2018
10:12 AM
samj1912
started around 11 hrs ago
2018-07-02 18309, 2018
10:13 AM
samj1912
*9
2018-07-02 18337, 2018
10:13 AM
ruaok
oh, perfect. yes. all normal.
2018-07-02 18351, 2018
10:13 AM
ruaok
but that graph should be updated. that doesn't relate to solr at all.
2018-07-02 18359, 2018
10:13 AM
ruaok
just gets us all sketchy...
2018-07-02 18317, 2018
10:14 AM
samj1912
yeah, its just a rmq graph
2018-07-02 18325, 2018
10:14 AM
samj1912
we should add a sir filter to it
2018-07-02 18358, 2018
10:14 AM
samj1912 is still confused about solr timings
2018-07-02 18345, 2018
10:51 AM
D4RK-PH0ENiX has quit
2018-07-02 18319, 2018
10:52 AM
D4RK-PH0ENiX joined the channel
2018-07-02 18335, 2018
10:56 AM
D4RK-PH0ENiX has quit
2018-07-02 18338, 2018
10:57 AM
ruaok
yvanzo & zas: invoices please!
2018-07-02 18325, 2018
11:09 AM
yokel has quit
2018-07-02 18354, 2018
11:10 AM
iliekcomputers
Leo_Verto: ping
2018-07-02 18343, 2018
11:11 AM
yokel joined the channel
2018-07-02 18359, 2018
11:11 AM
iliekcomputers
2018-07-02 18341, 2018
11:12 AM
iliekcomputers
Also, do you think we should just fix the unit tests in the PR and merge this and integration tests should be a different PR?
2018-07-02 18315, 2018
11:15 AM
D4RK-PH0ENiX joined the channel
2018-07-02 18317, 2018
11:19 AM
D4RK-PH0ENiX has quit
2018-07-02 18334, 2018
11:19 AM
D4RK-PH0ENiX joined the channel
2018-07-02 18346, 2018
11:41 AM
rsh7 joined the channel
2018-07-02 18335, 2018
12:00 PM
yokel has quit
2018-07-02 18314, 2018
12:03 PM
yokel joined the channel
2018-07-02 18325, 2018
12:08 PM
Leo__Verto joined the channel
2018-07-02 18326, 2018
12:15 PM
Leo__Verto has quit
