10:25 AM
zas
solr2 on its way back
2018-06-15 16607, 2018
10:27 AM
samj1912
That just means it won't distribute requests to other nodes
2018-06-15 16628, 2018
10:27 AM
samj1912
Well, there aren't nodes to distribute though :p
2018-06-15 16635, 2018
10:27 AM
samj1912
Solr will still work
2018-06-15 16610, 2018
10:28 AM
samj1912
Cloud graph working now
2018-06-15 16652, 2018
10:28 AM
samj1912
zas, can you turn solr3 back on?
2018-06-15 16637, 2018
10:34 AM
samj1912
zas?
2018-06-15 16626, 2018
10:35 AM
UmkaDK_ joined the channel
2018-06-15 16657, 2018
10:35 AM
ruaok
want me do to it?
2018-06-15 16634, 2018
10:36 AM
ruaok
solr-3 is coming back
2018-06-15 16659, 2018
10:36 AM
samj1912
Thanks :)
2018-06-15 16639, 2018
10:37 AM
UmkaDK has quit
2018-06-15 16606, 2018
10:38 AM
samj1912
solr3 in recovery mode
2018-06-15 16613, 2018
10:38 AM
samj1912
Missing tlog being played
2018-06-15 16617, 2018
10:38 AM
bukwurm has quit
2018-06-15 16636, 2018
10:39 AM
iliekcomputers
2018-06-15 16600, 2018
10:42 AM
samj1912
Recovered
2018-06-15 16610, 2018
10:44 AM
samj1912
Everything back to normal
2018-06-15 16621, 2018
10:44 AM
samj1912
ruaok, wanna turn on lb1 too?
2018-06-15 16635, 2018
10:44 AM
ruaok
done.
2018-06-15 16638, 2018
10:44 AM
ruaok
iliekcomputers: pondering.
2018-06-15 16655, 2018
10:45 AM
D4RK-PH0ENiX has quit
2018-06-15 16613, 2018
10:46 AM
D4RK-PH0ENiX joined the channel
2018-06-15 16653, 2018
10:46 AM
D4RK-PH0ENiX has quit
2018-06-15 16627, 2018
10:47 AM
D4RK-PH0ENiX joined the channel
2018-06-15 16618, 2018
10:49 AM
samj1912
lb switch working perfectly too
2018-06-15 16658, 2018
10:51 AM
D4RK-PH0ENiX has quit
2018-06-15 16601, 2018
10:52 AM
zas
yup, i see ruaok found the power on button ;) thanks, i was providing food to my little monster
2018-06-15 16620, 2018
10:52 AM
zas
and everything works smoothly it seems
2018-06-15 16650, 2018
10:52 AM
zas
coffee now, then we'll start to configure for beta test
2018-06-15 16602, 2018
10:54 AM
zas
2018-06-15 16635, 2018
10:54 AM
samj1912
zas, let's test it out :p
2018-06-15 16637, 2018
10:54 AM
zas
i'd like to test that on test.mb.o
2018-06-15 16644, 2018
10:54 AM
samj1912
Ditto
2018-06-15 16644, 2018
10:58 AM
samj1912
zas: are you switching?
2018-06-15 16604, 2018
10:59 AM
samj1912
I think there might be problems with the http/https thing though
2018-06-15 16614, 2018
11:00 AM
samj1912
zas: you might as well switch
2018-06-15 16641, 2018
11:00 AM
samj1912
I have turned off docker solr since day before
2018-06-15 16647, 2018
11:00 AM
samj1912
2018-06-15 16647, 2018
11:00 AM
samj1912
doesn't work
2018-06-15 16631, 2018
11:05 AM
samj1912
2018-06-15 16611, 2018
11:06 AM
samj1912
restarted the website container just now
2018-06-15 16655, 2018
11:07 AM
samj1912
2018-06-15 16657, 2018
11:07 AM
samj1912
seems to work
2018-06-15 16629, 2018
11:09 AM
samj1912
2018-06-15 16641, 2018
11:09 AM
samj1912
okay, test.mb switched to solr-cloud
2018-06-15 16627, 2018
11:10 AM
samj1912
CatQuest: reosarevok if you wanna try it out
2018-06-15 16651, 2018
11:10 AM
zas
samj1912: can you stop the load testing script now? so we can see requests from test.mb.o in logs
2018-06-15 16655, 2018
11:10 AM
samj1912
okay
2018-06-15 16635, 2018
11:11 AM
samj1912
2018-06-15 16640, 2018
11:11 AM
samj1912
final logs from live feed
2018-06-15 16650, 2018
11:11 AM
samj1912
turned off now
2018-06-15 16618, 2018
11:14 AM
samj1912
zas: 0 reqs :P
2018-06-15 16625, 2018
11:14 AM
samj1912
no one uses test.mb :P
2018-06-15 16629, 2018
11:14 AM
zas
ofc ;)
2018-06-15 16602, 2018
11:15 AM
zas
those stats are including when we stopped lb & solr nodes right?
2018-06-15 16614, 2018
11:15 AM
samj1912
yup
2018-06-15 16618, 2018
11:15 AM
D4RK-PH0ENiX joined the channel
2018-06-15 16619, 2018
11:15 AM
zas
so basically we lost only few requests :)
2018-06-15 16625, 2018
11:15 AM
samj1912
yup
2018-06-15 16640, 2018
11:15 AM
zas
99.97% isn't bad
2018-06-15 16651, 2018
11:15 AM
samj1912
keepup percent?
2018-06-15 16637, 2018
11:16 AM
samj1912 makes a couple of reqs to test.mb
2018-06-15 16658, 2018
11:16 AM
samj1912
lol, it doesn't even show up on the graph :P
2018-06-15 16634, 2018
11:17 AM
samj1912
2018-06-15 16638, 2018
11:17 AM
samj1912
zas: ^ ?
2018-06-15 16649, 2018
11:17 AM
samj1912
lol :P
2018-06-15 16603, 2018
11:18 AM
zas
euh?
2018-06-15 16626, 2018
11:18 AM
samj1912
I made too many reqs through test.mb :P
2018-06-15 16634, 2018
11:18 AM
zas
np :)
2018-06-15 16654, 2018
11:18 AM
samj1912
so, how do we test this? :P
2018-06-15 16656, 2018
11:18 AM
samj1912
beta? :P
2018-06-15 16605, 2018
11:19 AM
samj1912
even that has like 1-2 reqs/s
2018-06-15 16635, 2018
11:19 AM
samj1912
meanwhile lets setup the security?
2018-06-15 16647, 2018
11:20 AM
zas
2018-06-15 16655, 2018
11:20 AM
zas
Jun 15 13:20:48 solr-lb-1 haproxy[1466]: 138.201.203.28:42882 [15/Jun/2018:13:20:48.565] http-in servers/mb-solr-1 0/0/0/76/76 200 30581 - - ---- 0/0/0/0/0 0/0 "GET /solr/recording/basic?q=rose&start=0&rows=25&wt=mbjson HTTP/1.1"
2018-06-15 16605, 2018
11:21 AM
zas
76ms
2018-06-15 16633, 2018
11:21 AM
zas
Jun 15 13:21:14 solr-lb-1 haproxy[1466]: 138.201.203.28:42948 [15/Jun/2018:13:21:14.863] http-in servers/mb-solr-2 0/0/0/97/97 200 30875 - - ---- 0/0/0/0/0 0/0 "GET /solr/recording/basic?q=rose&start=0&rows=25&wt=mbjson HTTP/1.1"
2018-06-15 16634, 2018
11:21 AM
samj1912
hmm
2018-06-15 16640, 2018
11:21 AM
zas
97ms on solr2
2018-06-15 16605, 2018
11:22 AM
Leo__Verto joined the channel
2018-06-15 16632, 2018
11:25 AM
samj1912
zas: how do you wanna do the access control?
2018-06-15 16617, 2018
11:26 AM
samj1912
also, are we resizing solr2 and solr3? I wanna point sir to the floating ip
2018-06-15 16638, 2018
11:26 AM
zas
i think we need to properly set up auth on solr to start with + firewalling
2018-06-15 16646, 2018
11:26 AM
zas
2018-06-15 16611, 2018
11:27 AM
zas
so we can keep authentified access (and IP restrictions)
2018-06-15 16651, 2018
11:27 AM
samj1912
okay
2018-06-15 16653, 2018
11:27 AM
zas
i'll disallow any direct access to solr nodes
2018-06-15 16654, 2018
11:27 AM
samj1912
let me try
2018-06-15 16659, 2018
11:27 AM
samj1912
wait
2018-06-15 16607, 2018
11:28 AM
zas
meanwhile i'll rescale solr2 & 3
2018-06-15 16619, 2018
11:28 AM
samj1912
let me first shut down sir if you are disallowing access
2018-06-15 16629, 2018
11:28 AM
samj1912
otherwise we might lose data
2018-06-15 16631, 2018
11:28 AM
zas
i'll do it later, first i rescale
2018-06-15 16643, 2018
11:28 AM
samj1912
okay
2018-06-15 16651, 2018
11:28 AM
zas
i shut down solr3 for rescaling
2018-06-15 16614, 2018
11:29 AM
samj1912
okay
2018-06-15 16623, 2018
11:30 AM
zas
rescaling in progress
2018-06-15 16600, 2018
11:31 AM
samj1912
let me know once its done, I will have to change mem config for solr3
2018-06-15 16653, 2018
11:39 AM
ruaok
iliekcomputers: I've been thinking about our loggging/sentry setup.
2018-06-15 16626, 2018
11:40 AM
ruaok
I think we should brainstorm our setup a little and think up more logging levels and then move all of our error messages to those new log levels.
2018-06-15 16644, 2018
11:40 AM
ruaok
for instance, there are problems from scripts that we need to address that are not urgent.
2018-06-15 16603, 2018
11:41 AM
ruaok
then we have urgent things like the site being down that we're not getting good notifications on.
2018-06-15 16611, 2018
11:41 AM
ruaok
and loads of general info logging.
2018-06-15 16619, 2018
11:43 AM
github joined the channel
2018-06-15 16619, 2018
11:43 AM
github
[listenbrainz-server] mayhem closed pull request #393: Improved Spotify Integration (master...improved-embedded-spotify)
https://git.io/vxzhm
2018-06-15 16619, 2018
11:43 AM
github has left the channel
2018-06-15 16625, 2018
11:43 AM
ruaok
2018-06-15 16635, 2018
11:43 AM
ruaok
I like the first option you proposed to fix the issue.
2018-06-15 16648, 2018
11:43 AM
ruaok
when you have a moment from your GSoC work, I would love a PR to make that happen.
2018-06-15 16613, 2018
11:46 AM
samj1912
okay its up
2018-06-15 16641, 2018
11:46 AM
samj1912
zas: wait 5 mins before bringing down solr2
2018-06-15 16657, 2018
11:46 AM
iliekcomputers
ruaok: the error levels are a good idea , I like it, we don't use warning or critical anywhere, I think.
2018-06-15 16602, 2018
11:48 AM
samj1912
okay zas you can resize solr2 now
2018-06-15 16607, 2018
11:48 AM
zas
ok
2018-06-15 16614, 2018
11:48 AM
yokel has quit
2018-06-15 16642, 2018
11:48 AM
chhavi_
ruaok: With my skills it will be more than a moment :p But yes, added to my list of to-do :)
2018-06-15 16653, 2018
11:48 AM
zas
samj1912: in progress
2018-06-15 16602, 2018
11:49 AM
zas
im' digging haproxy docs to tune config
2018-06-15 16603, 2018
11:49 AM
samj1912
👍
2018-06-15 16602, 2018
11:50 AM
zas
i wonder a thing, currently we have pure round robin, but i guess we could take advantage of having queries from same client sticking to one backend server, as it is likely to improve caches efficiency
2018-06-15 16628, 2018
11:50 AM
samj1912
that was what I was going to ask
2018-06-15 16638, 2018
11:50 AM
samj1912
sticky sessions would be nice
2018-06-15 16615, 2018
11:52 AM
zas
yes, but they involve cookies, and ...
2018-06-15 16634, 2018
11:52 AM
zas
let me try to see if it works to start with
2018-06-15 16602, 2018
11:53 AM
samj1912
a responsive balancer would be nice instead of pure round robin
2018-06-15 16605, 2018
11:54 AM
yokel joined the channel
2018-06-15 16635, 2018
11:58 AM
ruaok
chhavi_: maybe we can pester Monkey to help... He wants to help...
2018-06-15 16612, 2018
11:59 AM
ruaok
samj1912: the loads looked nicely balanced to me. What else do you need?
2018-06-15 16641, 2018
11:59 AM
samj1912
ruaok: the current stats are phony :P
2018-06-15 16645, 2018
11:59 AM
samj1912
0 reqs coming through
2018-06-15 16656, 2018
11:59 AM
samj1912
its just avging out last 1k reqs
2018-06-15 16630, 2018
12:00 PM
samj1912
we are just configuring the last bits of security for solr
2018-06-15 16635, 2018
12:00 PM
samj1912
and then we will deploy it on beta
2018-06-15 16640, 2018
12:00 PM
samj1912
its working on test.mb currently
2018-06-15 16655, 2018
12:01 PM
ruaok
I was talking about the ones from last night...
2018-06-15 16607, 2018
12:02 PM
samj1912
ah
2018-06-15 16623, 2018
12:13 PM
pingupingu joined the channel
2018-06-15 16633, 2018
12:13 PM
samj1912
zas: its back?
2018-06-15 16614, 2018
12:14 PM
samj1912
I resized the heap
2018-06-15 16647, 2018
12:14 PM
zas
yes, but i'll reboot it soon, after upgrades
2018-06-15 16652, 2018
12:14 PM
zas
do not work on it yet
2018-06-15 16649, 2018
12:15 PM
zas
can you do few searchs on test.mb.o ?
2018-06-15 16655, 2018
12:15 PM
zas
i'm testing something
2018-06-15 16623, 2018
12:17 PM
zas
few more please
2018-06-15 16626, 2018
12:17 PM
samj1912
okay
2018-06-15 16624, 2018
12:18 PM
samj1912
enough?
2018-06-15 16627, 2018
12:18 PM
zas
2018-06-15 16634, 2018
12:18 PM
zas
yes, sticking to solr2
2018-06-15 16657, 2018
12:18 PM
samj1912
the result cache should have some nice stats
