#metabrainz

[picard] phw opened pull request #900: setup: picard.in does not need to get installed (master...setup-fixes) https://git.io/vhP1p

samj1912: ping

zas, pong

i just did a test: rebooting solr3 ... and everything stopped, is the script still running?

samj1912: ?

Let me see

Also zas, please don't reboot solr

*solr1

It's getting live indexes from sir

yes, but how does it recover if it happens?

Once the load balancer is setup properly, it will depend on the load balancer switch times

zas: no errors on the script side

so most prolly a log rotation

okay, script restarted

okay, let me try restarting now

solr3 stopped

script still seems to be runing

yup working

was prolly just a log rotation, it happens around this time

still working

the errors are from solr3 stopping

now let me restart it

samj1912: my thoughts about vms: we can afford 3 cx51 + 2 cx 11 for less than the price of 2 px61-ssd current used as main search servers

oh okay

3 cx41's wont cut it because

3cx51s would be nice

i wonder why we are capped at 90r/s

because of solr2 and solr3, they can serve at max 30

and you had the reqs divided equally

I think it will be able to take more, now that solr1 is getting more

even still solr1 can keep up with at max 60 reqs/s at 100% cpu usage

at more, it just lags slightly behind and response times go to shit

still, i'm not able to explain all numbers yet

it can still serve them without error

how so?

check https://stats.metabrainz.org/d/T4MODrIiz/solr-c...

yeah?

timings are changing a lot after a restart

it will stabilize after a while

caches for solr3 are empty

you don't see ? https://stats.metabrainz.org/d/T4MODrIiz/solr-c...

timings are much better

what did you restart exactly ? the whole cluster or just solr3 ?

just solr3

it might just be because of the live feed script

explain what it does exactly ? timings are 8 times better

zas: https://github.com/samj1912/solr-load-test/blob...

reads off the access log and tries its best to keepup

sorry, but i don't understand how it explains such different timings

solr3 timings are almost divided by 10

mean session time was around 500ms now it's around 50ms

(which is much more correct imho)

well, possibly a solr leader rebalance on restarting solr3

also, solr basically sucks at recovering after high loads, so I have seen

that I am saying from looking at the stress test and the stats on the dashboard

might possibly just be the lb stats

does the load balancer affect the response times?

dunno, this is why i said i can't explain all numbers (yet)

I would like to try what happens when we switch the load balancer

like we saw yesterday

lb2 had seriously bad perf.

yes, i'll try

which is weird, because it's exactly the same (software + vm)

im off for lunch bbiaw

ok

zas, solr's load, cpu and memory are the same, but response times are different

Probably something on the script/lb side

There has been absolutely no change on the pure solr stats

Even cache hits are the same

The only thing different after the restart is the response timing stats

samj1912: did you stop lb1 ?

if no, that's due to hetzner maintenance (they emitted a warning for today, on those nodes)

Nope I didn't

it confirms lb1/2 switch is working btw

Don't even know the ip

because we are now on lb2

Okay

The response times again went bad

zas, ruaok we should push this out to beta atleast

It can easily take beta load

First thing to do for that is secure solr

And figure out how to deal with that upstream stuff, iirc there was some problem with https / address and we could only use consul dns

I'm all for pushing to beta, samj1912. do you know how to do it? if not, work with yvanzo to do it.

ruaok: I know what to change in the docker config but I am not sure how to deploy it

zas, can you help with the firewall and lb?

sorry, i was updating docs about lb setup

samj1912: what do you want to achieve ?

ruaok: too soon ;) we still need to secure a bunch of things, and tests aren't yet finished

we don't even have an host name yet ;)

let me create one to start with

just a test! :)

also there's no https

yeah, zas, basically a mirror of your setup for docker solr

a mirror ? where?

Firewall for the solr admin panel and only allow queries from allowed ips

I mean the same config that you created for docker solr

With the restrictions on query params and all

not sure it is possible with haproxy

Why so?

ok, first solr admin panel is on the same port as everything

Yeah

so firewalling isn't really an option

how is that secured usually ? auth?

No, I meant firewall from certain ips

huh? i think you mean the reverse, allowing access from only certain IPs then ;)

Yeah