#metabrainz

ruaok: Yes, we can do that :)

yvanzo, have you gotten around to running the script I sent you yet?

samj1912, yvanzo: i re-use musicbrainz_server_session cookie, with cookie prefix haproxy option

cool

it needs more tests though, not sure yet if the config is correct, still digging docs

but that wont help with sir posting docs right?

zas: I am shifting sir from solr1 to floating ip?

for post i have another config i'm working on, we can target one server

or should I wait?

zas: always 1 server wont be nice

wait, i finish the config, then we can test

we need to target the leader

that will result in least latency

otherwise solr forwards the request to leader

and then leader sends replication packets back to other replicas

but how can we get the leader IP in haproxy?

hmm, I dont think we can

not sure

well, it embeds lua

can we query solr to get the leader?

we might

its too complicated though

lets leave it to later

ok

can you restart the load testing script?

okay, live or stress?

stress it !

okay

that will be directly on the lb though

not via mbs

that's ok

i just want to be sure everything is still ok, and test with 3 xc51

hmm, it doesnt load balance

not even docker is safe https://kromtech.com/blog/security-center/crypt...

due to the sticky thing

samj1912: please stop the script, i'll change the config

okay

stats if you want

ok, start it again

okay

play it again, sam!

^^ https://musicbrainz.org/label/3ccd1270-99d3-4b8...

running

lol. meta memes.

meta meta-memes? :P

meta is what we do best

it is part of our name, afterall.

samj1912: let it run few mins (the script, not ruaok)

XD

requests are perfectly balanced, but not the timings... weird it is

perfectly balanced, as all things should be

zas: something to do with stat display?

:)

samj1912: nope, it is confirmed by logs

that's more related to caches imho

or the leader thing

let me know when to stop

never stop !

let's stress til it dies! no no ... no no limit !

samj1912: looks at cpu usage

samj1912: give the replay script a percentage that is higher than 100%.

(will need code changes, but still)

solr1 62% and others are at 100%

where doe sit break?

ruaok: we are doing a stress test currently

still 100% 200 responses

wait let me increase concurrency then

3x concurrency now

ok

115 req/s zero loss on this test

zas, I am stopping sir for a while

is solr-1 running some other task that slows things down?

ruaok: yup

sir prolly

sir

ah.

and replication packets to other nodes

from which host do you run the test samj1912 ?

stopped sir

herb?

someone please write a tool called m'aam.

yup

ok

sir is down

ok stress again !

its on

didnt stop the stress test

zas: how is it?

oh wait

let me reshuffle the reqs

otherwise caches

ok

peaking at 212 ops

restarted with shuffle

you can increase concurrency a bit ? i'd like it to throw some"enough! enough! i'll tell you all!"

its at 100 :P

"She can't handle any more captain!!!"

samj1912: make it go to 11!

er, 110!

I am not sure herb can :P

can you run the same test script from the db server on hetzner cloud ?

thats destroyed

playing mean now. :)

herb is is very good (usually)

it needs the logs, zas.

okay, 300 concurrency

what gives :

* :P

last test

at 300 concurrency

solr1 has timings twice better than others...

if it doesn't work, I will run it from another node

still unexplained

zas: its not been down since forever

for now, we lose no query at 220 queries/s

its got to have a lot of stuff on memory

not just caches

apart from the jvm, solr indexes are kept in memory

can you loop the test so it runs 10 times ? and i will power down solr2 in the middle

10 times?

I will have to figure out a doc size

going almost 300 ops now?

nice

btw, set this concurrency to 1000

herb doesn't care, <1 load

lol okay

imho we'll need to work on reducing latency

(btw, the bandwidth isn't counted between hetzner machines)

solr machines reached 50 load

evident

latency in what terms?

little we can do between the regular and cloud machines.

the time needed to answer ONE query, because even without load i have the feeling it could be better than >100ms

samj1912: step by step ;)

ah, I am not sure how I can improve solr anymore

still running ?

where can I see the response time graphs?

load >70

zas: stopped