#musicbrainz-devel

ianmcorvidae has changed the topic to: http://musicbrainz.org/#devel | Next meeting agenda: Reviews, Decisions: 4828, 4354, 2650, 4282, 3204, 1935 (ocharles), Atlassian OnDemand Jira hosting pro/cons (ian)

hello!@

morning

morning

Well, not here :)

everything ticking along ok?

reosarevok: http://www.total-knowledge.com/~ilya/mips/ugt.html

djce: how much work would it be to add lenny to http://stats.musicbrainz.org/webstats/nginx-rrd... ?

can't remember. At the very least I could clone that graph and make a lenny-only version.

that would certainly be sufficient

done

great! thanks a bunch :)

ocharles: had some fun with data in my /ws/2 json tests being slightly different from what I expected.

ocharles: the tests still get stuff from the running memcached, so if "production" data differs from t/sql/webservice.sql and has been recently viewed it will affect the tests >_<

warp: that's odd, because a test context uses a null cache

(yes, it took me way too long before I realized that was going on)

ocharles: hmm, odd. I'll see if I can consistently reproduce it.

ocharles: ok, it's not happening now. I guess I did change something else right before/after I restarted memcached. grmbl.

ocharles: can http://tickets.musicbrainz.org/browse/MBS-3957 be taken off beta, since it can't be fixed before the release anyway?

nikkimini: yep, one moment

hm. zero-padded minutes are new, aren't they?

nikkimini: ok, done

yay, thanks

"current transaction is aborted" is back

only on beta?

probably

hm

nikkimini: fixed

oh?

it seems to be when that process gets deadlocked it doesn't rollback the transaction

ah

Project musicbrainz-server_beta build #52: STILL FAILING in 17 min: http://ci.musicbrainz.org/job/musicbrainz-serve...

ollie: Revert "Merge branch 'mbs-3957' into beta"

nikkimini: ok, I've identified the 'transaction aborted' crap and am working on a proper fix for it

nikkimini: but it should probably wait for the next release due to freeze and all that

Freeze doesn't apply to fixing stuff in beta

(unless this is also wrong in the normal servers)

it probably should, imo

if something's broken and found in freeze, we should just revert it

of course, in this case we can't

but it's quite a big change, i'd like to give it a thorough round of testing

ocharles, 502s on work-artist

back to working after a few mins

instant or timeout?

Timeout

haha wow, master failed because beta was also running tests and they deadlocked :)

Project musicbrainz-server_master build #314: FIXED in 11 min: http://ci.musicbrainz.org/job/musicbrainz-serve...

And some more work-artist 502s

hm, looks like no

no, i haven't worked out how

totoro looks fine

hmm, not entirely fine...

LOG: process 8008 acquired ShareLock on transaction 475590401 after 383689.608 ms

ok, I think I know the problem

I think a process encountered a deadlock after it had managed to hold some locks. the deadlock subsequently aborted the transaction, but did not release the locks

weird

but because the exception is rethrow in the web site, the transaction doesn't get rolled back

hence the transaction is never closed, and the locks are never released

http://codereview.musicbrainz.org/r/2101/ may be the solution there

ah

it does sound like it, reading the description

it randomly fixed itself because the process reaper killed the process that had been running over 600 seconds

heh

trying to correlate data now to see if my assumption is correct

ta1

!

moin!

moin

the reaper killed 3 processes today.

you might want to check chatlogs

i dunno when you joined

I will in a bit.

'wonders if they added locks to the pg stats' is where the interesting stuff is from

I saw your mail.

that's only 20 lines or so

??q:q

feh.

cofffeee!

I upped the reaper to to 1200s now.

and now coffee

btw, we should probably just statement_timeout in postgresql.conf for this

instead of doing killing ourselves

furthermore... it can be set on a per session basis

so we could just set it for our web frontends, and have other sessions not have timeouts

that also lets us cap the timeout at 30s

i.e. once it's 502'd, don't interfere further :)

yes, please!

'Aug 1 16:15:50 totoro postgres[1541]: [8-1] FATAL: terminating connection due to administrator command'

Shoot, I was hoping to actually know what query it killed.

Correlating that against the process reaper shows it as <idle>, so I think it's idle in transaction

I'm not sure if statement_timeout will help with that

ocharles: https://github.com/metabrainz/sys-tools/commit/...

ruaok: comments added

k

oh, I got the impression you were only checking the last one.

ocharles: https://github.com/metabrainz/sys-tools/commit/...

ruaok: that looks good

20 seconds for limit?

er idle

yea, that sounds ok

actually

go with 40

so we can be sure a frontend should have killed it

or 31 if you want to go as small as possible

deployed with 40

i'm not sure this code is right actually

ruaok: uhh, scalar(localtime) > $idle doesn't make any sense :)

that just checks 'is now greater than 20 seconds'

heh

i think it'd be much easier to amend that SELECT to have "current_query = '<IDLE> in transaction' AS idle_transaction" and then check if $stuck->{idle_transaction}

agreed.

interval/date/time manipulation is hard, easier to just keep that in the database side of things

ruaok: i'll leave this with you, I'd like to get heading home now

one sec.