#metabrainz

solr2 on its way back

Well, there aren't nodes to distribute though :p

Solr will still work

Cloud graph working now

zas, can you turn solr3 back on?

zas?

want me do to it?

solr-3 is coming back

Thanks :)

solr3 in recovery mode

Missing tlog being played

ruaok: opinions on https://github.com/metabrainz/listenbrainz-serv... ?

Recovered

Everything back to normal

ruaok, wanna turn on lb1 too?

done.

iliekcomputers: pondering.

lb switch working perfectly too

yup, i see ruaok found the power on button ;) thanks, i was providing food to my little monster

and everything works smoothly it seems

coffee now, then we'll start to configure for beta test

yvanzo: what will happen if we put solr-cloud.mb.o in https://github.com/metabrainz/docker-server-con...

zas, let's test it out :p

i'd like to test that on test.mb.o

Ditto

zas: are you switching?

I think there might be problems with the http/https thing though

zas: you might as well switch

I have turned off docker solr since day before

doesn't work

zas: it works :D https://test.musicbrainz.org/search?query=a&...

restarted the website container just now

seems to work

okay, test.mb switched to solr-cloud

CatQuest: reosarevok if you wanna try it out

samj1912: can you stop the load testing script now? so we can see requests from test.mb.o in logs

okay

final logs from live feed

turned off now

zas: 0 reqs :P

no one uses test.mb :P

ofc ;)

those stats are including when we stopped lb & solr nodes right?

yup

so basically we lost only few requests :)

yup

99.97% isn't bad

keepup percent?

lol, it doesn't even show up on the graph :P

zas: ^ ?

lol :P

euh?

I made too many reqs through test.mb :P

np :)

so, how do we test this? :P

beta? :P

even that has like 1-2 reqs/s

meanwhile lets setup the security?

Jun 15 13:20:48 solr-lb-1 haproxy[1466]: 138.201.203.28:42882 [15/Jun/2018:13:20:48.565] http-in servers/mb-solr-1 0/0/0/76/76 200 30581 - - ---- 0/0/0/0/0 0/0 "GET /solr/recording/basic?q=rose&start=0&rows=25&wt=mbjson HTTP/1.1"

76ms

Jun 15 13:21:14 solr-lb-1 haproxy[1466]: 138.201.203.28:42948 [15/Jun/2018:13:21:14.863] http-in servers/mb-solr-2 0/0/0/97/97 200 30875 - - ---- 0/0/0/0/0 0/0 "GET /solr/recording/basic?q=rose&start=0&rows=25&wt=mbjson HTTP/1.1"

hmm

97ms on solr2

zas: how do you wanna do the access control?

also, are we resizing solr2 and solr3? I wanna point sir to the floating ip

i think we need to properly set up auth on solr to start with + firewalling

https://lucidworks.com/2017/04/14/securing-solr... <-- can you test this?

so we can keep authentified access (and IP restrictions)

okay

i'll disallow any direct access to solr nodes

let me try

wait

meanwhile i'll rescale solr2 & 3

let me first shut down sir if you are disallowing access

otherwise we might lose data

i'll do it later, first i rescale

okay

i shut down solr3 for rescaling

okay

rescaling in progress

let me know once its done, I will have to change mem config for solr3

iliekcomputers: I've been thinking about our loggging/sentry setup.

I think we should brainstorm our setup a little and think up more logging levels and then move all of our error messages to those new log levels.

for instance, there are problems from scripts that we need to address that are not urgent.

then we have urgent things like the site being down that we're not getting good notifications on.

and loads of general info logging.

[listenbrainz-server] mayhem closed pull request #393: Improved Spotify Integration (master...improved-embedded-spotify) https://git.io/vxzhm

chhavi_: on this PR: https://github.com/metabrainz/listenbrainz-serv...

I like the first option you proposed to fix the issue.

when you have a moment from your GSoC work, I would love a PR to make that happen.

okay its up

zas: wait 5 mins before bringing down solr2

ruaok: the error levels are a good idea , I like it, we don't use warning or critical anywhere, I think.

okay zas you can resize solr2 now

ok

ruaok: With my skills it will be more than a moment :p But yes, added to my list of to-do :)

samj1912: in progress

im' digging haproxy docs to tune config

👍

i wonder a thing, currently we have pure round robin, but i guess we could take advantage of having queries from same client sticking to one backend server, as it is likely to improve caches efficiency

that was what I was going to ask

sticky sessions would be nice

yes, but they involve cookies, and ...

let me try to see if it works to start with

a responsive balancer would be nice instead of pure round robin

chhavi_: maybe we can pester Monkey to help... He wants to help...

samj1912: the loads looked nicely balanced to me. What else do you need?

ruaok: the current stats are phony :P

0 reqs coming through

its just avging out last 1k reqs

we are just configuring the last bits of security for solr

and then we will deploy it on beta

its working on test.mb currently

I was talking about the ones from last night...

ah

zas: its back?

I resized the heap

yes, but i'll reboot it soon, after upgrades

do not work on it yet

can you do few searchs on test.mb.o ?

i'm testing something

few more please

okay

enough?

yes, sticking to solr2

the result cache should have some nice stats