#musicbrainz-devel

ah yes

I didn't update beta

BEGIN failed--compilation aborted at line 4.

CONTEXT: compilation of PL/Perl function "extract_path_value"

nikki: beta and that JSON::XS problem should be fixed

load down to 50.

tail -F /etc/service/mb_server-fastcgi/log/main/current | grep -A20 '\[error\]' | tai64nlocal # on asterix, pingu and astro is looking quiet

good

two "expected" time outs

(subscriptions)

totoro should be ready for me to futz with, right?

the parts for pingu are our for dellivery.

I will run the memory test on totoro while I am out.

best to wait until I leave work in 20 minutes

oh, won't be yet.

ok

I'm not ready to leave the house.

i'm still pulling bits and bobs off totoro as I bring baron up

ok, np.

its amazing watching baron struggle with this load.

totoro is a nice fucking machine.

yea

ruaok: postgres-rrd should be pointed at baron now. how did you see it was unhappy? I should verify that my changes worked :)

i do hope baron can get down to under count(cpu_cores) soon though

ruaok: I think we really need to get on setting up a virtual ip for our db though

or a 'master-db' hostname

warp: it was a nagios email alert.

ocharles: fer sure.

I'm going to go throw these logs tomorrow and enumerate the various ways all of this went wrong

also, the dump locks script should be turned off.

various bits of uncommited configuration, non-scripted deployments, etc

hm.

ok, looks like pgbouncer is up

so can someone drive proxies while I point front ends to talk to that?

gimme 2 minutes

ok, i'll just do it

i have to leave in 10 minutes, but I'll be mostly above ground

back

still want help?

all done

sorry. :(

np

hm, that's quite some load on baron.

so we're ok for now?

postgres-rrd is still complaining.

I just forwarded the email to you warp.

hrm, odd that you get the complaints and I don't.

ha. the load is down to 15

oh, cron. not nagios.

ok, I get those.

bbiab

ok

baron is at 6. nice. :)

:)

:/ 502

Hope it's an isolated one

hm. timezones.

500...

load is defintely higher now.

Oh well, I guess I should go back to writing minipatches

the CPU is pegged.

we should lower our global connection limit.

nikki: fewer errors now?

pingus mobo arrived.

I'm not seeing any right now

ruaok: the postgresql timezone on baron is different, which is breaking dave's scripts.

ah. go ahead and fix the timezone then.

ruaok: I set the server timezone to UTC, so I think that will fix itself after a postgres restart. in the mean time, I patched the scripts.

k, thx

(I will have to unpatch them later :)

I lowered the global limit to 3000, from 3500.

Lets see if that helps a bit.

so that affect website visitors, web service visitors, or both?

everyone, IIRC

btw, search doesn't seem to be updating on beta. label results are over a day old

nikki: I'll take a look.

yup, there are some null pointer exceptions in the log for ijabz to look at.

nikki: I will shut down the updaters for other (other is everything but releases and recordigs) and do a full re-index of those.

then re-enable them.

nikki: have you made a SEARCH ticket for this? (if so, I will attach the logs there, if not... we need to create one :)

Whatever that affects, my edits still aren't going through :(

(the global limit thingy)

oh wait

the exceptions I'm seeing are just beta search not being able to connect to totoro.

reosarevok: I lowered the limit to 2500.

not sure that is going to help you. :(

nikki: alright, I'm not going to touch this until totoro is back, I will point beta.musicbrainz.org at the real search servers for the time being.

baron is just plain CPU bound. :(

I suspect that using something like the new pingu as a temporary DB replacement is going give better results.

it has twice the cores baron does

nikki: beta.musicbrainz.org should be using the production search servers now.

thanks

ocharles: when you return.. I think we may want to make lolo a READ ONLY DB slave, so we can spread the load a little.

pgbouncer can help with that right?

ocharles: ping

ruaok: pong, free in ten minutes just on the phone atm

k

but I don't think pgbouncer can help with that

I know people use this configuration. how do we do that?

if you can look into that, that would be awesome.

I will head to DWNI to start on pingu and totoro.

ocharles: apparently the caa.org/beta/ urls don't work

ruaok: you need your code to know when it's only doing read only work

I wonder if we take the most popular ws/2 call and point it elsewhere? would that help?

ocharles: think about it.

I;m heading to DWNI

if you need me, use gtalk mayem@ plz

ruaok: i have thought about it, it's a hard problem :)

nikki: fixed.

warp: thanks

the easiest would be to run a seperate musicbrainz-server, point that at read only. and do the routing in nginx.

(if we can identify a certain class of webservice requests which are always read only)

ocharles: I guess an authenticated webservice request will need to write last_login_date

iirc, there's really only a couple ws calls that aren't read-only, right?

wouldn't all GETs be read-only?

(or however the column is called)

stuff like the isrc submission, etc.

nikki: quite possibly not if they're authenticated.

I mean, it's not like we have an editable webservice :)

I thought ian said we're not using last_login_date now?

oh, right, most queries can be authenticated to get user tags :/

kepstin-work: they can even be authenticated if the authentication isn't used for anything.

i suppose picard probably just authenticates all requests if you give it a user account, eh

kepstin-work: so just checking for inc=user-tags isn't enough.

if the last_login thing isn't updated, it shouldn't matter.

can you detect any headers added for authentication, maybe?

when he checked the other day, it hasn't been updating it since we released ngs

since all write calls need authentication

and if it's taken 2 years for anyone to notice, it doesn't seem very useful anyway :P

yes, I think we can route based on /ws/2 + no authentication headers. in theory.

nikki: it was just put to use in the whole password reset fuss.

I said "not very useful", not "not at all useful" :P

last_login_date is updated since the password problems

I do not acknoledge this theory of usefulness not being binary ;)

anyway, yes, it's obviously easy to say "just make non-authenticated GETS load balance" but I'd like a stricter heuristic

warp: that's your problem then :P

I'd /really/ like to have decent effect typing in the data layer to say "this is a read only transaction" which would fail at compile time if you try and write inside it

ocharles: right. but mb server is not written in haskell :)

warp: and that is why I haven't done any work on this :)