#musicbrainz-devel

lol @ xkcd

lol!

50* errors :(

ocharles-, warp, ianmcorvidae, getting 502s for just about everything.

Anyone knows why the hell we're having 50x on every single page?

No, and no one with access seems to be around to figure it out.

Well

Seems to be back for now

It's been in and out for me for a while.

Yeah, ok

Gone again now

*grumbles*

hello!

warp: PING

ack

nikki_: yep, we're aware of it. and it's even worse now apparantly.

I thought the redis stuff was supposed to stop it from ISEing like that :/

site is back.

nikki_: this is a different ISE

well, there's still a bunch of these "Can't use an undefined value as a HASH reference" ones

nikki_: the theory was that either: 1. memcached would lose sessions (it's a cache, not a datastore). 2. if connection to memcached was lost a new session was created, so still losing the session.

nikki_: which is why we switched to redis, because in redis is a datastore, and the connection handling is better

but redis ran out of filehandles (memcached has stuff to deal with this, and redis as well, but our super old version of redis doesn't)

nikki_: and ofcourse there are many other things broken in the release editor which can make it ISE.

so I've noticed

the majority of the ISEs are the release editor crashing or people submitting cd stubs without a tracklist :(

oh, or the random search ones

Is it okay if I feel a great sense of satisfaction when I add a disc ID that kills a CD stub?

Leftmost: yes.

Just because you can't stop me doesn't mean it's okay. :-P

Hmmm, i cannot log to acoustid.org with my usual credentials, is this related to the issue MB just had ?

zas: might be

the MB auth requests are timing out

ohoho, 502 again on MB

warp: ping

ocharles: hello!

still file handle troubles?

ocharles: we've got redis at 100% cpu for no apparent reason

I don't think it's file handles.

what server?

roobarb

lets have a looksie

max open files is set correctly when I check /proc/19501/limits

connect clients hovers between 200 and 300 when I can connect. (redis-cli info)

connected

we have a 16 core machine so a load of 2 doesn't seem the end of the world, I guess?

yeah, that should be fine.

can I have sudo on that machine?

sure

ocharles: done.

thanks

hrm. now there's two redis-server's running.

redis seems to be writing 30MB/s

so I imagine that cpu usage is almost entirely io dominated

ok

so it needs to flush/save less.

but according to /var/log/redis it isn't flushing that ofte

at most it should flush once every minute. but only if 10000 keys have changed since the last save.

hum, mabye it's not that, atop isn't showing much activity for the disk actually

:(

ocharles: what are you currently doing?

(there's still two redis-servers running, which cannot be good, I'd like to kill one

)

i'm doing nothing

go ahead

redis 19501 64.5 9.2 1533664 1527524 ? Ss 12:05 24:17 /usr/bin/redis-server /etc/redis/redis.conf

redis 21088 25.8 9.2 1534108 1527768 ? R 12:38 1:17 /usr/bin/redis-server /etc/redis/redis.conf

root 18846 0.0 0.0 8292 720 pts/4 S+ 11:41 0:00 tail -f redis-server.log

acid2 21241 0.0 0.0 7628 1020 pts/6 S+ 12:44 0:00 grep --color=auto redis

oddly, I saw no servers

which did you kill?

19501, which also seems to have killed the other one. so perhaps it's normal.

the logs say:

30 Mar 12:04:55 * Opening TCP port: bind: Address already in use

but if one is an intentional/internal fork of the other, I would have expected their start time to be the same.

so I don't think it's normal

also

30 Mar 12:44:47 - The server is now ready to accept connections on port 6379

30 Mar 11:20:14 * WARNING overcommit_memory is set to 0! Background save may fail under low condition memory. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.

but that's fail, not spin

the server has 4GB free memory, so that message doesn't seem relevant to our current trouble.

this connection is too weak to do anything useful :/

ocharles: anyway, so I don't understand why it's being this finnicky. it should have enough file handles, enough CPU, enough memory.

acid2 [at roobarb]:~$ ps aux | grep strc

a^C^[[A^C^C^C^C^Z

for example

:)

ecuador cable internet ftw!

ocharles: I'm inclined to install redis 2.x either on roobarb or a new hoser vm. 1.2 seems old. though upgrading and hoping that magically fixes a problem is in general not the best strategy.

i'm ok with that

ok

i really can't do anything other than advise i'm afraid

i'm seeing java take 400% of the cpu though atm

ocharles: yep, roobarb and dora are the search servers.

i know

but i mean the load doesn't seem to be coming from redis right now

and redis is working fine when it's not spiking at 100%

it seems to have only broken today though

and ruaok did an upgrade on the search servers yesterday, so i wonder if the events are correlated?

http://stats.musicbrainz.org/webstats/nginx-rrd... shows it kicked in around 1am

though that upgrade looks to have finished around 3 hours before

ocharles: and 3 hours is the interval at which we deploy search indexes? :)

i thought we did that in a loop now

then the loop takes 3 hours? or I'm just misremembering.

3 hours is what it says on the /search page

Search Results

Last updated: 2013-03-30 08:34 GMT

that does seem to be quite a while ago

that is certainly more than 3 hours.

yea

ok, I've got a redis 2.6 on dora.

cool

shall I just switch things over, or should I make some attempt at preserving the sessions?

uff, roobarb at load 13 again

and no, just switch them

alright.

it seems to be java that's really being problematic here

then switching to dora isn't going to help. it currently has java at 400% and load 4.something.