#musicbrainz-devel

/

      • nikki_ joined the channel
      • 2012-08-01 21446, 2012

      • voiceinsideyou joined the channel
      • 2012-08-01 21431, 2012

      • flamingspinach joined the channel
      • 2012-08-01 21403, 2012

      • ruaok joined the channel
      • 2012-08-01 21403, 2012

      • nikkimini joined the channel
      • 2012-08-01 21405, 2012

      • kepstin-laptop joined the channel
      • 2012-08-01 21428, 2012

      • Prophet5 joined the channel
      • 2012-08-01 21447, 2012

      • ianmcorvidae
        ianmcorvidae has changed the topic to: http://musicbrainz.org/#devel | Next meeting agenda: Reviews, Decisions: 4828, 4354, 2650, 4282, 3204, 1935 (ocharles), Atlassian OnDemand Jira hosting pro/cons (ian)
      • 2012-08-01 21446, 2012

      • ijabz joined the channel
      • 2012-08-01 21445, 2012

      • Mineo joined the channel
      • 2012-08-01 21440, 2012

      • reosarevok joined the channel
      • 2012-08-01 21443, 2012

      • ijabz joined the channel
      • 2012-08-01 21440, 2012

      • warp
        hello!@
      • 2012-08-01 21429, 2012

      • ijabz joined the channel
      • 2012-08-01 21436, 2012

      • djce joined the channel
      • 2012-08-01 21408, 2012

      • ocharles
        morning
      • 2012-08-01 21427, 2012

      • reosarevok
        morning
      • 2012-08-01 21430, 2012

      • reosarevok
        Well, not here :)
      • 2012-08-01 21409, 2012

      • ocharles
        everything ticking along ok?
      • 2012-08-01 21411, 2012

      • warp
      • 2012-08-01 21437, 2012

      • ocharles
        djce: how much work would it be to add lenny to http://stats.musicbrainz.org/webstats/nginx-rrd/d… ?
      • 2012-08-01 21400, 2012

      • stefans joined the channel
      • 2012-08-01 21417, 2012

      • djce
        can't remember. At the very least I could clone that graph and make a lenny-only version.
      • 2012-08-01 21406, 2012

      • ocharles
        that would certainly be sufficient
      • 2012-08-01 21450, 2012

      • djce
        done
      • 2012-08-01 21418, 2012

      • noobie joined the channel
      • 2012-08-01 21402, 2012

      • ocharles
        great! thanks a bunch :)
      • 2012-08-01 21441, 2012

      • ijabz joined the channel
      • 2012-08-01 21413, 2012

      • ijabz_ joined the channel
      • 2012-08-01 21441, 2012

      • warp *facepalm*
      • 2012-08-01 21450, 2012

      • warp
        ocharles: had some fun with data in my /ws/2 json tests being slightly different from what I expected.
      • 2012-08-01 21442, 2012

      • warp
        ocharles: the tests still get stuff from the running memcached, so if "production" data differs from t/sql/webservice.sql and has been recently viewed it will affect the tests >_<
      • 2012-08-01 21401, 2012

      • ocharles
        warp: that's odd, because a test context uses a null cache
      • 2012-08-01 21407, 2012

      • warp
        (yes, it took me way too long before I realized that was going on)
      • 2012-08-01 21429, 2012

      • warp
        ocharles: hmm, odd. I'll see if I can consistently reproduce it.
      • 2012-08-01 21448, 2012

      • warp
        ocharles: ok, it's not happening now. I guess I did change something else right before/after I restarted memcached. grmbl.
      • 2012-08-01 21402, 2012

      • voiceinsideyou joined the channel
      • 2012-08-01 21412, 2012

      • reosarevok joined the channel
      • 2012-08-01 21447, 2012

      • djce joined the channel
      • 2012-08-01 21412, 2012

      • nikkimini
        ocharles: can http://tickets.musicbrainz.org/browse/MBS-3957 be taken off beta, since it can't be fixed before the release anyway?
      • 2012-08-01 21411, 2012

      • ocharles
        nikkimini: yep, one moment
      • 2012-08-01 21438, 2012

      • nikkimini
        hm. zero-padded minutes are new, aren't they?
      • 2012-08-01 21442, 2012

      • ocharles
        nikkimini: ok, done
      • 2012-08-01 21419, 2012

      • nikkimini
        yay, thanks
      • 2012-08-01 21403, 2012

      • nikkimini
        "current transaction is aborted" is back
      • 2012-08-01 21439, 2012

      • ocharles
        only on beta?
      • 2012-08-01 21451, 2012

      • nikkimini
        probably
      • 2012-08-01 21409, 2012

      • ocharles
        hm
      • 2012-08-01 21422, 2012

      • ocharles
        nikkimini: fixed
      • 2012-08-01 21434, 2012

      • nikkimini
        oh?
      • 2012-08-01 21435, 2012

      • ocharles
        it seems to be when that process gets deadlocked it doesn't rollback the transaction
      • 2012-08-01 21445, 2012

      • nikkimini
        ah
      • 2012-08-01 21409, 2012

      • MBJenkins
        Project musicbrainz-server_beta build #52: STILL FAILING in 17 min: http://ci.musicbrainz.org/job/musicbrainz-server_…
      • 2012-08-01 21410, 2012

      • MBJenkins
        ollie: Revert "Merge branch 'mbs-3957' into beta"
      • 2012-08-01 21431, 2012

      • ocharles
        nikkimini: ok, I've identified the 'transaction aborted' crap and am working on a proper fix for it
      • 2012-08-01 21428, 2012

      • ocharles
        nikkimini: but it should probably wait for the next release due to freeze and all that
      • 2012-08-01 21414, 2012

      • reosarevok
        Freeze doesn't apply to fixing stuff in beta
      • 2012-08-01 21426, 2012

      • reosarevok
        (unless this is also wrong in the normal servers)
      • 2012-08-01 21429, 2012

      • ocharles
        it probably should, imo
      • 2012-08-01 21433, 2012

      • voiceinsideyou joined the channel
      • 2012-08-01 21436, 2012

      • ocharles
        if something's broken and found in freeze, we should just revert it
      • 2012-08-01 21446, 2012

      • ocharles
        of course, in this case we can't
      • 2012-08-01 21456, 2012

      • ocharles
        but it's quite a big change, i'd like to give it a thorough round of testing
      • 2012-08-01 21441, 2012

      • reosarevok
        ocharles, 502s on work-artist
      • 2012-08-01 21418, 2012

      • reosarevok
        back to working after a few mins
      • 2012-08-01 21419, 2012

      • ocharles
        instant or timeout?
      • 2012-08-01 21406, 2012

      • reosarevok
        Timeout
      • 2012-08-01 21458, 2012

      • ocharles
        haha wow, master failed because beta was also running tests and they deadlocked :)
      • 2012-08-01 21448, 2012

      • MBJenkins
        Project musicbrainz-server_master build #314: FIXED in 11 min: http://ci.musicbrainz.org/job/musicbrainz-server_…
      • 2012-08-01 21437, 2012

      • kepstin-work joined the channel
      • 2012-08-01 21446, 2012

      • voiceinsideyou joined the channel
      • 2012-08-01 21401, 2012

      • voiceinsideyou1 joined the channel
      • 2012-08-01 21412, 2012

      • voiceinsideyou2 joined the channel
      • 2012-08-01 21405, 2012

      • hawke_ joined the channel
      • 2012-08-01 21409, 2012

      • Leftmost joined the channel
      • 2012-08-01 21419, 2012

      • reosarevok
        And some more work-artist 502s
      • 2012-08-01 21450, 2012

      • ianmcorvidae wonders if they added locks to the pg stats
      • 2012-08-01 21423, 2012

      • ianmcorvidae
        hm, looks like no
      • 2012-08-01 21456, 2012

      • ocharles
        no, i haven't worked out how
      • 2012-08-01 21415, 2012

      • ocharles
        totoro looks fine
      • 2012-08-01 21434, 2012

      • ocharles
        hmm, not entirely fine...
      • 2012-08-01 21438, 2012

      • ocharles
        LOG: process 8008 acquired ShareLock on transaction 475590401 after 383689.608 ms
      • 2012-08-01 21451, 2012

      • ruaok joined the channel
      • 2012-08-01 21422, 2012

      • ocharles
        ok, I think I know the problem
      • 2012-08-01 21410, 2012

      • ocharles
        I think a process encountered a deadlock after it had managed to hold some locks. the deadlock subsequently aborted the transaction, but did not release the locks
      • 2012-08-01 21423, 2012

      • ianmcorvidae
        weird
      • 2012-08-01 21430, 2012

      • ocharles
        but because the exception is rethrow in the web site, the transaction doesn't get rolled back
      • 2012-08-01 21439, 2012

      • ocharles
        hence the transaction is never closed, and the locks are never released
      • 2012-08-01 21453, 2012

      • ocharles
      • 2012-08-01 21438, 2012

      • ianmcorvidae
        ah
      • 2012-08-01 21404, 2012

      • ianmcorvidae
        it does sound like it, reading the description
      • 2012-08-01 21424, 2012

      • ocharles
        it randomly fixed itself because the process reaper killed the process that had been running over 600 seconds
      • 2012-08-01 21434, 2012

      • ianmcorvidae
        heh
      • 2012-08-01 21434, 2012

      • plaintext joined the channel
      • 2012-08-01 21438, 2012

      • ocharles
        trying to correlate data now to see if my assumption is correct
      • 2012-08-01 21446, 2012

      • ianmcorvidae reviews that bit of code, one way or the other
      • 2012-08-01 21453, 2012

      • ocharles
        ta1
      • 2012-08-01 21454, 2012

      • ocharles
        !
      • 2012-08-01 21409, 2012

      • ocharles
        it makes me quite uneasy, that review. but tests do pass...
      • 2012-08-01 21419, 2012

      • ruaok
        moin!
      • 2012-08-01 21422, 2012

      • ocharles
        moin
      • 2012-08-01 21427, 2012

      • ruaok
        the reaper killed 3 processes today.
      • 2012-08-01 21435, 2012

      • ocharles
        you might want to check chatlogs
      • 2012-08-01 21437, 2012

      • ocharles
        i dunno when you joined
      • 2012-08-01 21454, 2012

      • ruaok
        I will in a bit.
      • 2012-08-01 21458, 2012

      • ocharles
        'wonders if they added locks to the pg stats' is where the interesting stuff is from
      • 2012-08-01 21400, 2012

      • ruaok
        I saw your mail.
      • 2012-08-01 21401, 2012

      • ocharles
        that's only 20 lines or so
      • 2012-08-01 21428, 2012

      • ruaok
        ??q:q
      • 2012-08-01 21431, 2012

      • ruaok
        feh.
      • 2012-08-01 21434, 2012

      • ruaok
        cofffeee!
      • 2012-08-01 21443, 2012

      • ruaok
        I upped the reaper to to 1200s now.
      • 2012-08-01 21449, 2012

      • ruaok
        and now coffee
      • 2012-08-01 21453, 2012

      • ocharles
        btw, we should probably just statement_timeout in postgresql.conf for this
      • 2012-08-01 21457, 2012

      • ocharles
        instead of doing killing ourselves
      • 2012-08-01 21422, 2012

      • ocharles
        furthermore... it can be set on a per session basis
      • 2012-08-01 21436, 2012

      • ocharles
        so we could just set it for our web frontends, and have other sessions not have timeouts
      • 2012-08-01 21453, 2012

      • ocharles
        that also lets us cap the timeout at 30s
      • 2012-08-01 21427, 2012

      • ianmcorvidae
        i.e. once it's 502'd, don't interfere further :)
      • 2012-08-01 21432, 2012

      • ianmcorvidae likes that idea
      • 2012-08-01 21407, 2012

      • ruaok
        yes, please!
      • 2012-08-01 21458, 2012

      • ocharles finishes reading serverlog
      • 2012-08-01 21435, 2012

      • ocharles
        'Aug 1 16:15:50 totoro postgres[1541]: [8-1] FATAL: terminating connection due to administrator command'
      • 2012-08-01 21443, 2012

      • ocharles
        Shoot, I was hoping to actually know what query it killed.
      • 2012-08-01 21457, 2012

      • ocharles
        Correlating that against the process reaper shows it as <idle>, so I think it's idle in transaction
      • 2012-08-01 21419, 2012

      • ocharles
        I'm not sure if statement_timeout will help with that
      • 2012-08-01 21421, 2012

      • ruaok
      • 2012-08-01 21443, 2012

      • ocharles
        ruaok: comments added
      • 2012-08-01 21448, 2012

      • ruaok
        k
      • 2012-08-01 21417, 2012

      • ruaok
        oh, I got the impression you were only checking the last one.
      • 2012-08-01 21415, 2012

      • thewordsmith joined the channel
      • 2012-08-01 21402, 2012

      • ruaok
      • 2012-08-01 21446, 2012

      • ocharles
        ruaok: that looks good
      • 2012-08-01 21455, 2012

      • ruaok
        20 seconds for limit?
      • 2012-08-01 21458, 2012

      • ruaok
        er idle
      • 2012-08-01 21410, 2012

      • ocharles
        yea, that sounds ok
      • 2012-08-01 21417, 2012

      • ocharles
        actually
      • 2012-08-01 21419, 2012

      • ocharles
        go with 40
      • 2012-08-01 21429, 2012

      • ocharles
        so we can be sure a frontend should have killed it
      • 2012-08-01 21446, 2012

      • ocharles
        or 31 if you want to go as small as possible
      • 2012-08-01 21456, 2012

      • ruaok
        deployed with 40
      • 2012-08-01 21457, 2012

      • ocharles
        i'm not sure this code is right actually
      • 2012-08-01 21426, 2012

      • ocharles
        ruaok: uhh, scalar(localtime) > $idle doesn't make any sense :)
      • 2012-08-01 21434, 2012

      • ocharles
        that just checks 'is now greater than 20 seconds'
      • 2012-08-01 21417, 2012

      • ruaok
        heh
      • 2012-08-01 21442, 2012

      • ocharles
        i think it'd be much easier to amend that SELECT to have "current_query = '<IDLE> in transaction' AS idle_transaction" and then check if $stuck->{idle_transaction}
      • 2012-08-01 21451, 2012

      • ruaok
        agreed.
      • 2012-08-01 21405, 2012

      • ocharles
        interval/date/time manipulation is hard, easier to just keep that in the database side of things
      • 2012-08-01 21404, 2012

      • ocharles
        ruaok: i'll leave this with you, I'd like to get heading home now
      • 2012-08-01 21410, 2012

      • ruaok
        one sec.