#metabrainz

oh sorry

running from another node

wait let me stop that too

we reached 400 ops

that was from the gcloud machine

now let's see if solr cloud recovers

I'm looking at the that page, but I can't make out which graph, zas.

from the diff. test

concurrency was 500

samj1912: ok, enough, let it cool down

we are above our target of 270 req/s

yay :D

this should improve with live logs

and if we need more, we can just add a node...

more cache hits

what was the worse latency during stress?

cx11 for haproxy are perfect, cpu load was still low

and memory is enough

dunno, i'll analyze graphs in a few

we had a weird peak on solr3

?

but worse figure is ~500ms

nice

that too when it was HAMMERED

which is acceptable for that load ;)

anyway, we are good, now we have to secure this stuff

phew.

sign off from the boss. wooo!

please document everything, i'll update docs with last changes i made to haproxy conf

but first, cofffffeee

Leo__Verto: not yet. Note that even email domains cannot go public, except for email hosting domains.

is it okay if I filter those manually or should I find a list and have the script filter by that?

samj1912: can you target sir at solr-cloud.musicbrainz.org ?

i set it up to forward requests to solr1 only for now, i want to see if it works as expected (on POST)

zas: instead I will try posting from my local machine

and just annotations

whatever ;)

all valid requests should start with /solr/ path right?

zas: your change to test mb json is perfect, the same will do for beta.

zas: yes

zas: done

there should have been 4 requests

it works, all went to solr1

zas: what happens when solr1 is down?

I never thought that would happen

:)

lol

great!

zas: retried with a bigger collection

zas: how was it?

good, try again

i changed few things

solr2 and 3 are set as backups, so if solr1 is down, they'll be used for POST

okay

reposted

ok it still works, now stop solr1 and retry

okay

solr1 stopped

zas: how was it reposted?

yes, on solr2

so it works as expected, you can restart solr1

I wanna see what happens when I stop it in between

like two kids in a sandbox trying to break their toys.

<3

better now, when it will be in prod that will not be as fun ;)

true

okay ,currently solr2 is leader

reposting URL (4.5 million large), lets see how it handles url changing

zas: rather can you power solr2 down when I ask?

sure

I think that stopping it manually will let it play some replication packets and clear the index before shutting down

okay

tell me when

okay zas, as soon as you see reqs to solr-2

stop it

now

zas:

they all go to solr1 due to config

but lets see if it works

and if they recover properly

solr2 stopped

oh, they are not going to solr2?

nope, due to config, solr2/3 are set as backups of solr1 for now

ah

wait 1 second then

I stopped solr3 as well

lets see what happens

solr1 returns 503

thats coz solr1 forwards it to solr3 which was the current leader

so it didn't take the leadership

we need at least 2 servers up

(as said, over and over;)

starting solr2

solr3 is back

solr1 has leadership

solr2 returning

yes, all 3 are up

now let me retry with solr1 now that it has leadership

ok

solr2 is recovering

url still recovering

i see no query atm

for solr2 . no

others will reply

i mean for all

oh no, others reply

they do??

if solr2 is recovering, in case a query comes, it sends it to other nodes

yeah

but no query on lb

because no one is querying? :P

:)

Leo__Verto: I blanked email domains appearing less than 10 times.

okay recovered

now let me play it to solr1

ah yeah, that works too

and bring solr1 down

zas: shutdown solr1 please

when I say

samj1912: sure

zas: now

samj1912: i have an idea for that, we coud write a special health check script

done, solr1 going down

zas: did reqs go to any other node?

or all end up on solr1?

basically if we can get the current leader, we could just mark it as healthy, and others as unhealthy

solr2

okay, so partly to solr1 and partly to solr2?

yup

cool

solr3 was the leader after solr1 went down

lets bring solr1 back up

there were few 504s

any 5xx?

4

okay

hmm

expected since that's not instant

I will add some retries to pysolr

lets tackle security next

samj1912: i think we should just round robin on each node for POST, and don't bother with leader thing

sounds sane.

after all, it's solr cloud matter