-
D4RK-PH0ENiX has quit
2018-06-15 16652, 2018
-
Leo__Verto has quit
2018-06-15 16659, 2018
-
D4RK-PH0ENiX joined the channel
2018-06-15 16616, 2018
-
Leo__Verto joined the channel
2018-06-15 16602, 2018
-
Leo__Verto has quit
2018-06-15 16645, 2018
-
drsaunders has quit
2018-06-15 16608, 2018
-
G0re joined the channel
2018-06-15 16627, 2018
-
Gore|woerk has quit
2018-06-15 16616, 2018
-
pingupingu joined the channel
2018-06-15 16616, 2018
-
pingupingu has quit
2018-06-15 16616, 2018
-
pingupingu joined the channel
2018-06-15 16657, 2018
-
pingupingu has quit
2018-06-15 16652, 2018
-
pingupingu joined the channel
2018-06-15 16610, 2018
-
pingupingu has quit
2018-06-15 16633, 2018
-
slurpee- joined the channel
2018-06-15 16635, 2018
-
slurpee- has quit
2018-06-15 16614, 2018
-
Slurpee has quit
2018-06-15 16632, 2018
-
pingupingu joined the channel
2018-06-15 16603, 2018
-
pingupingu has quit
2018-06-15 16637, 2018
-
pingupingu joined the channel
2018-06-15 16609, 2018
-
Gazooo has quit
2018-06-15 16649, 2018
-
Gazooo joined the channel
2018-06-15 16628, 2018
-
rsh7 joined the channel
2018-06-15 16605, 2018
-
outsidecontext joined the channel
2018-06-15 16626, 2018
-
github joined the channel
2018-06-15 16626, 2018
-
github
[picard] phw opened pull request #900: setup: picard.in does not need to get installed (master...setup-fixes)
https://git.io/vhP1p
2018-06-15 16626, 2018
-
github has left the channel
2018-06-15 16656, 2018
-
Monkey joined the channel
2018-06-15 16617, 2018
-
Monkey has quit
2018-06-15 16649, 2018
-
zas
samj1912: ping
2018-06-15 16615, 2018
-
samj1912
zas, pong
2018-06-15 16654, 2018
-
zas
i just did a test: rebooting solr3 ... and everything stopped, is the script still running?
2018-06-15 16600, 2018
-
thomasross has quit
2018-06-15 16612, 2018
-
thomasross joined the channel
2018-06-15 16610, 2018
-
zas
samj1912: ?
2018-06-15 16621, 2018
-
samj1912
Let me see
2018-06-15 16619, 2018
-
samj1912
Also zas, please don't reboot solr
2018-06-15 16624, 2018
-
samj1912
*solr1
2018-06-15 16642, 2018
-
samj1912
It's getting live indexes from sir
2018-06-15 16658, 2018
-
zas
yes, but how does it recover if it happens?
2018-06-15 16611, 2018
-
samj1912
Once the load balancer is setup properly, it will depend on the load balancer switch times
2018-06-15 16607, 2018
-
samj1912
zas: no errors on the script side
2018-06-15 16615, 2018
-
samj1912
so most prolly a log rotation
2018-06-15 16612, 2018
-
samj1912
okay, script restarted
2018-06-15 16623, 2018
-
Leo__Verto joined the channel
2018-06-15 16659, 2018
-
samj1912
okay, let me try restarting now
2018-06-15 16652, 2018
-
samj1912
solr3 stopped
2018-06-15 16610, 2018
-
samj1912
script still seems to be runing
2018-06-15 16624, 2018
-
samj1912
yup working
2018-06-15 16638, 2018
-
samj1912
was prolly just a log rotation, it happens around this time
2018-06-15 16651, 2018
-
samj1912
2018-06-15 16653, 2018
-
samj1912
still working
2018-06-15 16600, 2018
-
samj1912
the errors are from solr3 stopping
2018-06-15 16612, 2018
-
samj1912
now let me restart it
2018-06-15 16650, 2018
-
Nyanko-sensei joined the channel
2018-06-15 16617, 2018
-
zas
samj1912: my thoughts about vms: we can afford 3 cx51 + 2 cx 11 for less than the price of 2 px61-ssd current used as main search servers
2018-06-15 16639, 2018
-
samj1912
oh okay
2018-06-15 16649, 2018
-
samj1912
3 cx41's wont cut it because
2018-06-15 16658, 2018
-
samj1912
3cx51s would be nice
2018-06-15 16616, 2018
-
zas
i wonder why we are capped at 90r/s
2018-06-15 16635, 2018
-
samj1912
because of solr2 and solr3, they can serve at max 30
2018-06-15 16642, 2018
-
samj1912
and you had the reqs divided equally
2018-06-15 16656, 2018
-
D4RK-PH0ENiX has quit
2018-06-15 16605, 2018
-
samj1912
I think it will be able to take more, now that solr1 is getting more
2018-06-15 16629, 2018
-
samj1912
even still solr1 can keep up with at max 60 reqs/s at 100% cpu usage
2018-06-15 16648, 2018
-
samj1912
at more, it just lags slightly behind and response times go to shit
2018-06-15 16650, 2018
-
zas
still, i'm not able to explain all numbers yet
2018-06-15 16653, 2018
-
samj1912
it can still serve them without error
2018-06-15 16655, 2018
-
samj1912
how so?
2018-06-15 16655, 2018
-
zas
2018-06-15 16613, 2018
-
samj1912
yeah?
2018-06-15 16617, 2018
-
zas
timings are changing a lot after a restart
2018-06-15 16637, 2018
-
samj1912
it will stabilize after a while
2018-06-15 16644, 2018
-
samj1912
caches for solr3 are empty
2018-06-15 16643, 2018
-
zas
2018-06-15 16653, 2018
-
zas
timings are much better
2018-06-15 16626, 2018
-
zas
what did you restart exactly ? the whole cluster or just solr3 ?
2018-06-15 16636, 2018
-
samj1912
just solr3
2018-06-15 16644, 2018
-
samj1912
it might just be because of the live feed script
2018-06-15 16640, 2018
-
zas
explain what it does exactly ? timings are 8 times better
2018-06-15 16604, 2018
-
samj1912
2018-06-15 16622, 2018
-
samj1912
reads off the access log and tries its best to keepup
2018-06-15 16641, 2018
-
zas
sorry, but i don't understand how it explains such different timings
2018-06-15 16610, 2018
-
zas
solr3 timings are almost divided by 10
2018-06-15 16635, 2018
-
zas
mean session time was around 500ms now it's around 50ms
2018-06-15 16646, 2018
-
zas
(which is much more correct imho)
2018-06-15 16657, 2018
-
SothoTalKer has quit
2018-06-15 16628, 2018
-
samj1912
well, possibly a solr leader rebalance on restarting solr3
2018-06-15 16647, 2018
-
samj1912
also, solr basically sucks at recovering after high loads, so I have seen
2018-06-15 16609, 2018
-
SothoTalKer joined the channel
2018-06-15 16649, 2018
-
samj1912
that I am saying from looking at the stress test and the stats on the dashboard
2018-06-15 16658, 2018
-
samj1912
might possibly just be the lb stats
2018-06-15 16653, 2018
-
samj1912
does the load balancer affect the response times?
2018-06-15 16640, 2018
-
zas
dunno, this is why i said i can't explain all numbers (yet)
2018-06-15 16601, 2018
-
Monkey joined the channel
2018-06-15 16608, 2018
-
samj1912
I would like to try what happens when we switch the load balancer
2018-06-15 16614, 2018
-
samj1912
like we saw yesterday
2018-06-15 16621, 2018
-
samj1912
lb2 had seriously bad perf.
2018-06-15 16624, 2018
-
zas
yes, i'll try
2018-06-15 16643, 2018
-
zas
which is weird, because it's exactly the same (software + vm)
2018-06-15 16646, 2018
-
samj1912
im off for lunch bbiaw
2018-06-15 16654, 2018
-
zas
ok
2018-06-15 16627, 2018
-
pingupingu has quit
2018-06-15 16655, 2018
-
Nyanko-sensei has quit
2018-06-15 16632, 2018
-
D4RK-PH0ENiX joined the channel
2018-06-15 16644, 2018
-
samj1912
zas, solr's load, cpu and memory are the same, but response times are different
2018-06-15 16656, 2018
-
samj1912
Probably something on the script/lb side
2018-06-15 16612, 2018
-
Leo__Verto has quit
2018-06-15 16615, 2018
-
samj1912
There has been absolutely no change on the pure solr stats
2018-06-15 16621, 2018
-
samj1912
Even cache hits are the same
2018-06-15 16637, 2018
-
samj1912
The only thing different after the restart is the response timing stats
2018-06-15 16658, 2018
-
zas
samj1912: did you stop lb1 ?
2018-06-15 16648, 2018
-
zas
if no, that's due to hetzner maintenance (they emitted a warning for today, on those nodes)
2018-06-15 16614, 2018
-
samj1912
Nope I didn't
2018-06-15 16616, 2018
-
zas
it confirms lb1/2 switch is working btw
2018-06-15 16621, 2018
-
samj1912
Don't even know the ip
2018-06-15 16625, 2018
-
zas
because we are now on lb2
2018-06-15 16633, 2018
-
samj1912
Okay
2018-06-15 16602, 2018
-
samj1912
The response times again went bad
2018-06-15 16642, 2018
-
samj1912
zas, ruaok we should push this out to beta atleast
2018-06-15 16650, 2018
-
samj1912
It can easily take beta load
2018-06-15 16630, 2018
-
samj1912
First thing to do for that is secure solr
2018-06-15 16625, 2018
-
samj1912
And figure out how to deal with that upstream stuff, iirc there was some problem with https / address and we could only use consul dns
2018-06-15 16641, 2018
-
ruaok
I'm all for pushing to beta, samj1912. do you know how to do it? if not, work with yvanzo to do it.
2018-06-15 16650, 2018
-
samj1912
ruaok: I know what to change in the docker config but I am not sure how to deploy it
2018-06-15 16621, 2018
-
samj1912
zas, can you help with the firewall and lb?
2018-06-15 16658, 2018
-
suhas2go has quit
2018-06-15 16633, 2018
-
zas
sorry, i was updating docs about lb setup
2018-06-15 16645, 2018
-
zas
samj1912: what do you want to achieve ?
2018-06-15 16649, 2018
-
ruaok is half tempted to switch production search to solr cloud for an hour and see if anyone notices
2018-06-15 16614, 2018
-
zas
ruaok: too soon ;) we still need to secure a bunch of things, and tests aren't yet finished
2018-06-15 16631, 2018
-
zas
we don't even have an host name yet ;)
2018-06-15 16645, 2018
-
zas
let me create one to start with
2018-06-15 16650, 2018
-
ruaok
just a test! :)
2018-06-15 16650, 2018
-
zas
also there's no https
2018-06-15 16620, 2018
-
samj1912
yeah, zas, basically a mirror of your setup for docker solr
2018-06-15 16648, 2018
-
zas
a mirror ? where?
2018-06-15 16648, 2018
-
samj1912
Firewall for the solr admin panel and only allow queries from allowed ips
2018-06-15 16627, 2018
-
samj1912
I mean the same config that you created for docker solr
2018-06-15 16649, 2018
-
samj1912
With the restrictions on query params and all
2018-06-15 16630, 2018
-
zas
not sure it is possible with haproxy
2018-06-15 16648, 2018
-
samj1912
Why so?
2018-06-15 16651, 2018
-
zas
ok, first solr admin panel is on the same port as everything
2018-06-15 16603, 2018
-
samj1912
Yeah
2018-06-15 16618, 2018
-
zas
so firewalling isn't really an option
2018-06-15 16630, 2018
-
zas
how is that secured usually ? auth?
2018-06-15 16636, 2018
-
samj1912
No, I meant firewall from certain ips
2018-06-15 16604, 2018
-
samj1912
You can just allow access to it from our hetzner ips
2018-06-15 16610, 2018
-
zas
huh? i think you mean the reverse, allowing access from only certain IPs then ;)
2018-06-15 16617, 2018
-
samj1912
Yeah
