-
chhavi_
ruaok: Yes, we can do that :)
2018-06-15 16659, 2018
-
Leo__Verto
yvanzo, have you gotten around to running the script I sent you yet?
2018-06-15 16638, 2018
-
zas
samj1912, yvanzo: i re-use musicbrainz_server_session cookie, with cookie prefix haproxy option
2018-06-15 16614, 2018
-
samj1912
cool
2018-06-15 16627, 2018
-
zas
it needs more tests though, not sure yet if the config is correct, still digging docs
2018-06-15 16645, 2018
-
samj1912
but that wont help with sir posting docs right?
2018-06-15 16612, 2018
-
samj1912
zas: I am shifting sir from solr1 to floating ip?
2018-06-15 16615, 2018
-
zas
for post i have another config i'm working on, we can target one server
2018-06-15 16617, 2018
-
samj1912
or should I wait?
2018-06-15 16628, 2018
-
samj1912
zas: always 1 server wont be nice
2018-06-15 16629, 2018
-
zas
wait, i finish the config, then we can test
2018-06-15 16632, 2018
-
samj1912
we need to target the leader
2018-06-15 16641, 2018
-
samj1912
that will result in least latency
2018-06-15 16650, 2018
-
samj1912
otherwise solr forwards the request to leader
2018-06-15 16604, 2018
-
samj1912
and then leader sends replication packets back to other replicas
2018-06-15 16613, 2018
-
zas
but how can we get the leader IP in haproxy?
2018-06-15 16625, 2018
-
samj1912
hmm, I dont think we can
2018-06-15 16631, 2018
-
samj1912
not sure
2018-06-15 16632, 2018
-
zas
well, it embeds lua
2018-06-15 16648, 2018
-
zas
can we query solr to get the leader?
2018-06-15 16653, 2018
-
samj1912
we might
2018-06-15 16659, 2018
-
samj1912
its too complicated though
2018-06-15 16603, 2018
-
samj1912
lets leave it to later
2018-06-15 16624, 2018
-
zas
ok
2018-06-15 16621, 2018
-
zas
can you restart the load testing script?
2018-06-15 16603, 2018
-
samj1912
okay, live or stress?
2018-06-15 16608, 2018
-
zas
stress it !
2018-06-15 16620, 2018
-
samj1912
okay
2018-06-15 16629, 2018
-
samj1912
that will be directly on the lb though
2018-06-15 16634, 2018
-
samj1912
not via mbs
2018-06-15 16635, 2018
-
zas
that's ok
2018-06-15 16657, 2018
-
zas
i just want to be sure everything is still ok, and test with 3 xc51
2018-06-15 16627, 2018
-
zas
hmm, it doesnt load balance
2018-06-15 16635, 2018
-
Leo__Verto
2018-06-15 16635, 2018
-
zas
due to the sticky thing
2018-06-15 16658, 2018
-
zas
samj1912: please stop the script, i'll change the config
2018-06-15 16610, 2018
-
samj1912
okay
2018-06-15 16627, 2018
-
samj1912
2018-06-15 16630, 2018
-
samj1912
stats if you want
2018-06-15 16622, 2018
-
zas
ok, start it again
2018-06-15 16645, 2018
-
samj1912
okay
2018-06-15 16602, 2018
-
ruaok
play it again, sam!
2018-06-15 16608, 2018
-
ruaok runs away
2018-06-15 16637, 2018
-
zas
2018-06-15 16640, 2018
-
samj1912
running
2018-06-15 16652, 2018
-
ruaok
lol. meta memes.
2018-06-15 16610, 2018
-
samj1912
meta meta-memes? :P
2018-06-15 16613, 2018
-
Leo__Verto
meta is what we do best
2018-06-15 16627, 2018
-
ruaok
it is part of our name, afterall.
2018-06-15 16641, 2018
-
zas
samj1912: let it run few mins (the script, not ruaok)
2018-06-15 16603, 2018
-
samj1912
XD
2018-06-15 16639, 2018
-
zas
requests are perfectly balanced, but not the timings... weird it is
2018-06-15 16652, 2018
-
Leo__Verto
perfectly balanced, as all things should be
2018-06-15 16600, 2018
-
Leo__Verto just couldn't resist
2018-06-15 16601, 2018
-
samj1912
zas: something to do with stat display?
2018-06-15 16602, 2018
-
zas
:)
2018-06-15 16615, 2018
-
zas
samj1912: nope, it is confirmed by logs
2018-06-15 16633, 2018
-
zas
that's more related to caches imho
2018-06-15 16602, 2018
-
zas
or the leader thing
2018-06-15 16605, 2018
-
samj1912
let me know when to stop
2018-06-15 16610, 2018
-
zas
never stop !
2018-06-15 16648, 2018
-
ruaok feels like he stumbled into an inspirational hippie IRC channel.
2018-06-15 16650, 2018
-
zas
let's stress til it dies! no no ... no no limit !
2018-06-15 16652, 2018
-
zas
samj1912: looks at cpu usage
2018-06-15 16601, 2018
-
ruaok
samj1912: give the replay script a percentage that is higher than 100%.
2018-06-15 16608, 2018
-
ruaok
(will need code changes, but still)
2018-06-15 16608, 2018
-
zas
solr1 62% and others are at 100%
2018-06-15 16612, 2018
-
ruaok
where doe sit break?
2018-06-15 16630, 2018
-
samj1912
ruaok: we are doing a stress test currently
2018-06-15 16641, 2018
-
zas
still 100% 200 responses
2018-06-15 16659, 2018
-
samj1912
wait let me increase concurrency then
2018-06-15 16618, 2018
-
samj1912
2018-06-15 16643, 2018
-
samj1912
3x concurrency now
2018-06-15 16647, 2018
-
zas
ok
2018-06-15 16608, 2018
-
zas
115 req/s zero loss on this test
2018-06-15 16610, 2018
-
samj1912
zas, I am stopping sir for a while
2018-06-15 16611, 2018
-
ruaok
is solr-1 running some other task that slows things down?
2018-06-15 16621, 2018
-
samj1912
ruaok: yup
2018-06-15 16621, 2018
-
zas
sir prolly
2018-06-15 16624, 2018
-
samj1912
sir
2018-06-15 16629, 2018
-
ruaok
ah.
2018-06-15 16639, 2018
-
samj1912
and replication packets to other nodes
2018-06-15 16643, 2018
-
zas
from which host do you run the test samj1912 ?
2018-06-15 16643, 2018
-
samj1912
stopped sir
2018-06-15 16645, 2018
-
zas
herb?
2018-06-15 16646, 2018
-
ruaok
someone please write a tool called m'aam.
2018-06-15 16648, 2018
-
samj1912
yup
2018-06-15 16651, 2018
-
zas
ok
2018-06-15 16621, 2018
-
samj1912
sir is down
2018-06-15 16631, 2018
-
zas
ok stress again !
2018-06-15 16639, 2018
-
samj1912
its on
2018-06-15 16647, 2018
-
samj1912
didnt stop the stress test
2018-06-15 16624, 2018
-
samj1912
zas: how is it?
2018-06-15 16639, 2018
-
samj1912
oh wait
2018-06-15 16645, 2018
-
samj1912
let me reshuffle the reqs
2018-06-15 16647, 2018
-
samj1912
otherwise caches
2018-06-15 16651, 2018
-
zas
ok
2018-06-15 16607, 2018
-
samj1912
2018-06-15 16611, 2018
-
zas
peaking at 212 ops
2018-06-15 16649, 2018
-
samj1912
restarted with shuffle
2018-06-15 16659, 2018
-
zas
you can increase concurrency a bit ? i'd like it to throw some"enough! enough! i'll tell you all!"
2018-06-15 16619, 2018
-
samj1912
its at 100 :P
2018-06-15 16625, 2018
-
ruaok
"She can't handle any more captain!!!"
2018-06-15 16635, 2018
-
ruaok
samj1912: make it go to 11!
2018-06-15 16638, 2018
-
ruaok
er, 110!
2018-06-15 16640, 2018
-
samj1912
I am not sure herb can :P
2018-06-15 16646, 2018
-
zas
can you run the same test script from the db server on hetzner cloud ?
2018-06-15 16600, 2018
-
samj1912
thats destroyed
2018-06-15 16605, 2018
-
ruaok
playing mean now. :)
2018-06-15 16610, 2018
-
zas
herb is is very good (usually)
2018-06-15 16621, 2018
-
ruaok
it needs the logs, zas.
2018-06-15 16644, 2018
-
samj1912
okay, 300 concurrency
2018-06-15 16648, 2018
-
samj1912
what gives :
2018-06-15 16650, 2018
-
samj1912
* :P
2018-06-15 16605, 2018
-
samj1912
2018-06-15 16607, 2018
-
samj1912
last test
2018-06-15 16647, 2018
-
samj1912
at 300 concurrency
2018-06-15 16659, 2018
-
zas
solr1 has timings twice better than others...
2018-06-15 16600, 2018
-
samj1912
if it doesn't work, I will run it from another node
2018-06-15 16612, 2018
-
zas
still unexplained
2018-06-15 16621, 2018
-
samj1912
zas: its not been down since forever
2018-06-15 16628, 2018
-
zas
for now, we lose no query at 220 queries/s
2018-06-15 16630, 2018
-
samj1912
its got to have a lot of stuff on memory
2018-06-15 16640, 2018
-
samj1912
not just caches
2018-06-15 16604, 2018
-
samj1912
apart from the jvm, solr indexes are kept in memory
2018-06-15 16652, 2018
-
zas
can you loop the test so it runs 10 times ? and i will power down solr2 in the middle
2018-06-15 16613, 2018
-
samj1912
10 times?
2018-06-15 16622, 2018
-
samj1912
I will have to figure out a doc size
2018-06-15 16634, 2018
-
samj1912
going almost 300 ops now?
2018-06-15 16652, 2018
-
zas
yes
2018-06-15 16606, 2018
-
samj1912
nice
2018-06-15 16643, 2018
-
zas
btw, set this concurrency to 1000
2018-06-15 16653, 2018
-
zas
herb doesn't care, <1 load
2018-06-15 16659, 2018
-
samj1912
lol okay
2018-06-15 16627, 2018
-
zas
imho we'll need to work on reducing latency
2018-06-15 16605, 2018
-
zas
(btw, the bandwidth isn't counted between hetzner machines)
2018-06-15 16635, 2018
-
zas
solr machines reached 50 load
2018-06-15 16637, 2018
-
samj1912
2018-06-15 16648, 2018
-
samj1912
evident
2018-06-15 16648, 2018
-
ruaok
latency in what terms?
2018-06-15 16659, 2018
-
ruaok
little we can do between the regular and cloud machines.
2018-06-15 16616, 2018
-
samj1912 just wants to put it in prod now
2018-06-15 16625, 2018
-
zas
the time needed to answer ONE query, because even without load i have the feeling it could be better than >100ms
2018-06-15 16652, 2018
-
zas
samj1912: step by step ;)
2018-06-15 16613, 2018
-
samj1912
ah, I am not sure how I can improve solr anymore
2018-06-15 16648, 2018
-
zas
still running ?
2018-06-15 16651, 2018
-
ruaok
where can I see the response time graphs?
2018-06-15 16653, 2018
-
zas
load >70
2018-06-15 16600, 2018
-
samj1912
zas: stopped
2018-06-15 16603, 2018
-
zas
