#musicbrainz-devel

search server is timing out again...

rest of the site's being slow as well. just got a 502 bad gateway.

we have unhappy servers

started right about 1/2 hour ago.

next time things to bad, please txt me!

things should be better now.

last time I tried to text it didn't seem to go through

cool, thx

hmm.

so we know that the jvm isn't the problem

it could be tomcat or the search server code.

fuss

what are you doing to "fix" this problem

is it worth having an hourly server restart in cron?

well, investigating the problem first of all.

(akin to the constant restarting we have in mb_server)

well, possibly.

its clearly a case of something running out of filehandles.

near 7000 open files things to belly up.

lsof show you what's open?

restarting tomcat works

i think it's lsof anyway

I've just been doing netstat | wc -l

we know who has the files open.

I'm mainly interested in finding out how many are open.

ok, but what are the files? do more files open on every search request?

and it not a clear case of file handles leaking straight out.

the number goes up, then drops.

but over time, it creeps upwards.

it happens faster with only one server as with two.

as in one server leaks faster than 2?

and, I think, with our increased traffic we're hitting a problem that we didn't hit before.

thus is coming around to bite us now.

yes.

I'm guessing that ever X requests our servers tip over.

hmm

X requests per server.

so with one server you hit twice as fast as with two servers.

it must be a X requests/N duration, not just X requests

and restarting the server isn't as easy as it is with the mb_server.

wait, I don't know that for sure.

we do a restart on the server every 3 hours.

but I am not sure a restart is enough to free the handles.

it may need to be stop/start

X/N might also make sense.

do I have any access to these servers? just curious what lsof shows

let me see if dora has calmed down now.

7235

100

what'd I do? :)

the first one was when dora was tits up.

the second 5 minutes after diverting traffic.

and doing nothing?

so, sockets are being kept open too long and eventually we run out of open file handles.

yea

that sounds like a reasonable hypothesis

they are all in CLOSE_WAIT status

or the garbage collection on sockets is not fast enough

oh wait, so the problem is that the search server is running out of sockets, not file handles?

so that would indicate that musicbrainz-server isn't closing it's connections properly

the cool down after 5 minutes is very likely from the musicbrainz-server fastcgi processes being killed and forced to close sockets

on astro, see /tmp

lsof output for both dora and rhubarb.

dora is idle.

roobarb is live.

sockets == filehandles.

yes! that is a great observation.

perhaps this is the fault of mb_server.

well, we should see a high open socket count on the mb-server machines too, no?

i know sockets == file handles, but I'm not sure if lsof shows them

hmm, but doesn't astro talk to carl before dora?

yes.

nginx on carl is doing the load balancing.

so, wouldn't carl have problems first?

hard to say off hand.

could be carl, could be carl propagating problems via mb_server

ruaok: hrm, keep dora out

maybe if I try and locally do some requests from my box to dora, we can see if the socket count goes up at all?

rhubarb will suffer the same problem.

ah.

we can try.

make the requests from hobbes

ok

which ip?

dora.mb will work :)

port though?

8080

ok, 1 request a second atm

erm, one sec :)

ok, off

this is using curl to do the requests from bash

straight to dora

was 30 open, now 19.

no seeming effect.

i'll up the rate then

and next we should try going through the load balancer

yea

ok, twice as fast now

well, the load balancer is going go to both, so thats not so good.

19

up the rate much higher

10 requests a second now

now 20

anything interesting?

20 open handles.

try 1000 requests per sec.

ok

:-)

30

that look like it's actually getting a lot of traffic in the logs too?

we don't actually record traffic on these servers.

ah, ok

1.63 load.

so doing somehting

double it. :)

"while [ 1 -eq 1 ]; do curl -s http://dora.mb:8080/ws/2/artist/?query=Hello > /dev/null& sleep 0.001; done" is my test :)

load went up to 0.64 on hobbes just doing that

lol

next text can be through carl

then we can test direct to dora with LWP

and the through carl with LWP

not bouncing up at all.

but I wonder if we should leave it running longer.

ruaok: I think a better option would be: while [ 1 -eq 1 ] do; netstat | wc -l >> sockets; sleep 30; done on dora

and bring it back into production

get a very rough idea of how that count changes over time

yeah.

want an account on dora?

sure

gimme a sec.

mom is whipping up a storm of activity

geez, she's not sick anymore

driving me insane.

ready!

heh

syntax error near done. :-(

dora is back in.

you get it running?

no

the count is climbing fast though.

while [ 1 -eq 1 ]; do netstat | wc -l; sleep 1; done >> counter

redirection in the wrong place, my bad

I'm going to take dora out and let things calm down.