#metabrainz

/

      • loujine has quit
      • 2016-08-11 22400, 2016

      • loujine joined the channel
      • 2016-08-11 22452, 2016

      • Mineo has quit
      • 2016-08-11 22417, 2016

      • Lotheric has quit
      • 2016-08-11 22438, 2016

      • Lotheric joined the channel
      • 2016-08-11 22452, 2016

      • ariscop joined the channel
      • 2016-08-11 22411, 2016

      • Lotheric_ joined the channel
      • 2016-08-11 22429, 2016

      • Lotheric has quit
      • 2016-08-11 22447, 2016

      • D4RK-PH0ENiX has quit
      • 2016-08-11 22416, 2016

      • D4RK-PH0ENiX joined the channel
      • 2016-08-11 22441, 2016

      • D4RK-PH0ENiX has quit
      • 2016-08-11 22403, 2016

      • pingupingu joined the channel
      • 2016-08-11 22432, 2016

      • D4RK-PH0ENiX joined the channel
      • 2016-08-11 22457, 2016

      • ZarkBit has quit
      • 2016-08-11 22447, 2016

      • D4RK-PH0ENiX has quit
      • 2016-08-11 22422, 2016

      • D4RK-PH0ENiX joined the channel
      • 2016-08-11 22434, 2016

      • D4RK-PH0ENiX has quit
      • 2016-08-11 22428, 2016

      • D4RK-PH0ENiX joined the channel
      • 2016-08-11 22453, 2016

      • pingupingu has quit
      • 2016-08-11 22430, 2016

      • mrc0mmand has quit
      • 2016-08-11 22419, 2016

      • pingupingu joined the channel
      • 2016-08-11 22443, 2016

      • mrc0mmand joined the channel
      • 2016-08-11 22417, 2016

      • JonnyJD joined the channel
      • 2016-08-11 22457, 2016

      • Kode_ joined the channel
      • 2016-08-11 22439, 2016

      • Kode has quit
      • 2016-08-11 22439, 2016

      • flamingspinach has quit
      • 2016-08-11 22444, 2016

      • Mineo joined the channel
      • 2016-08-11 22412, 2016

      • flamingspinach joined the channel
      • 2016-08-11 22444, 2016

      • Gore joined the channel
      • 2016-08-11 22427, 2016

      • rahulr has quit
      • 2016-08-11 22404, 2016

      • rahulr joined the channel
      • 2016-08-11 22404, 2016

      • rahulr has quit
      • 2016-08-11 22404, 2016

      • rahulr joined the channel
      • 2016-08-11 22428, 2016

      • Mineo has quit
      • 2016-08-11 22407, 2016

      • bitmap
        sitemaps have been failing 'cause of the meb api 504ing since july 24 https://gist.github.com/mwiencek/780c0df49f5d4207…
      • 2016-08-11 22426, 2016

      • bitmap
        not every day, just every few days
      • 2016-08-11 22409, 2016

      • bitmap
        was never a problem before that though
      • 2016-08-11 22452, 2016

      • bitmap
        updated the gist with the timestamp of every occurence, if that helps narrow down a problem
      • 2016-08-11 22443, 2016

      • rvedotrc has quit
      • 2016-08-11 22405, 2016

      • rvedotrc joined the channel
      • 2016-08-11 22419, 2016

      • ruaok
        yeah, we've been seeing random 504s. they are not just limited to one site, which suggests that it is a gateway problem
      • 2016-08-11 22420, 2016

      • bitmap
        ah
      • 2016-08-11 22442, 2016

      • bitmap
        I'll add some code to retry to request there anyway
      • 2016-08-11 22430, 2016

      • pingupingu has quit
      • 2016-08-11 22425, 2016

      • pingupingu joined the channel
      • 2016-08-11 22425, 2016

      • pingupingu has quit
      • 2016-08-11 22425, 2016

      • pingupingu joined the channel
      • 2016-08-11 22414, 2016

      • UmkaDK has quit
      • 2016-08-11 22441, 2016

      • UmkaDK joined the channel
      • 2016-08-11 22429, 2016

      • ruaok
        Gentlecat: does the sentry setup catch 504s for meb on sakura?
      • 2016-08-11 22428, 2016

      • reosarevok is in Spain now, will be somewhat more active :)
      • 2016-08-11 22432, 2016

      • reosarevok
        Still no mobile internet, mind
      • 2016-08-11 22444, 2016

      • reosarevok answered all the emails for now
      • 2016-08-11 22405, 2016

      • reosarevok sighs at 5 days of subscription edit mails :D
      • 2016-08-11 22421, 2016

      • Gentlecat
        ruaok: of its in the app, it should
      • 2016-08-11 22425, 2016

      • Gentlecat
        if*
      • 2016-08-11 22437, 2016

      • ruaok
        ok, then it idn
      • 2016-08-11 22443, 2016

      • ruaok
        isn't sakura's fault.
      • 2016-08-11 22408, 2016

      • drsaunders has quit
      • 2016-08-11 22449, 2016

      • ZarkBit joined the channel
      • 2016-08-11 22422, 2016

      • D4RK-PH0ENiX has quit
      • 2016-08-11 22435, 2016

      • D4RK-PH0ENiX joined the channel
      • 2016-08-11 22440, 2016

      • D4RK-PH0ENiX has quit
      • 2016-08-11 22407, 2016

      • D4RK-PH0ENiX joined the channel
      • 2016-08-11 22400, 2016

      • adhawkins
        Morning. Just came in to see if you were interested in another 504 report, but it seems like you already know about them :)
      • 2016-08-11 22405, 2016

      • ariscop has quit
      • 2016-08-11 22430, 2016

      • adhawkins
        For what it's worth, I had another at 05:16:02 downloading a replication packet.
      • 2016-08-11 22412, 2016

      • ruaok
        yeah, we know.
      • 2016-08-11 22444, 2016

      • ruaok
        at least we no know that it is a gateway problem and not a app server problem.
      • 2016-08-11 22419, 2016

      • ruaok
        and I'd hate to stick zas on it, because we should be focusing on moving to NewHost, but zas remains on vacation.
      • 2016-08-11 22448, 2016

      • ruaok
        so, shitty service will remain.
      • 2016-08-11 22447, 2016

      • ruaok desperately hopes that a time will come when he won't feel super frustrated about work
      • 2016-08-11 22406, 2016

      • adhawkins
        It's not so bad to be honest. I only notice it because occasionally I get a failure email from my slave server. It seems to recover itself the following hour.
      • 2016-08-11 22457, 2016

      • zas
        Replication related 504s are due to Sakura, they happen when it swaps heavily. I'm working on openresty container atm
      • 2016-08-11 22453, 2016

      • ruaok
        and why is sakura swapping?
      • 2016-08-11 22416, 2016

      • ruaok
        75 uwsgi processes running.
      • 2016-08-11 22418, 2016

      • ruaok
        gee, I wonder.
      • 2016-08-11 22415, 2016

      • ruaok
        adhawkins: let me know if it happens again
      • 2016-08-11 22435, 2016

      • ruaok
        zas: does graphana have a graph for examining memory usage for a given server?
      • 2016-08-11 22410, 2016

      • zas
        Yes server overview
      • 2016-08-11 22419, 2016

      • ruaok
        thx
      • 2016-08-11 22440, 2016

      • ruaok reboots sakura
      • 2016-08-11 22419, 2016

      • ruaok
        yay, metabrainz.org 500s now.
      • 2016-08-11 22444, 2016

      • ruaok
      • 2016-08-11 22447, 2016

      • ruaok
        sigh.
      • 2016-08-11 22441, 2016

      • ruaok
        I've got an idea.
      • 2016-08-11 22446, 2016

      • ruaok
        let's just shut everything down.
      • 2016-08-11 22451, 2016

      • ruaok
        bring it back up when we're ready.
      • 2016-08-11 22456, 2016

      • ruaok
        in a few months.
      • 2016-08-11 22400, 2016

      • ruaok
        much less stress that way.
      • 2016-08-11 22449, 2016

      • Kode_
        what did you decide on in the end for NewHost ruaok?
      • 2016-08-11 22413, 2016

      • ruaok
        hetzner. their deal can't be beat.
      • 2016-08-11 22441, 2016

      • Kode_
        yeah?
      • 2016-08-11 22442, 2016

      • ZarkBit has quit
      • 2016-08-11 22417, 2016

      • ruaok
        why is sakura such a freaking mess?
      • 2016-08-11 22441, 2016

      • Kode_
        I blame zas, and when it isn't his fault it's surely Fresos
      • 2016-08-11 22445, 2016

      • Kode_
        ;)
      • 2016-08-11 22423, 2016

      • ruaok
        yeah, hetzner doesn't charge us for VAT either, since we're a US company. there is 20% savings right there.
      • 2016-08-11 22454, 2016

      • ruaok
        and we got clarification on the terms and things are better than we first thought.
      • 2016-08-11 22409, 2016

      • ruaok
        the only real downside is that we need to rent space in a rack for future use.
      • 2016-08-11 22431, 2016

      • ruaok
        but even that sunk cost is considerably less than going with OVH.
      • 2016-08-11 22445, 2016

      • ruaok
        and Google Cloud is FOUR times as expensive as Hetzner.
      • 2016-08-11 22456, 2016

      • Kode_
        no brainer then
      • 2016-08-11 22424, 2016

      • ruaok
        yeah. and we know what we're getting with hetzner. we have an established relationship with them.
      • 2016-08-11 22440, 2016

      • ruaok
        we're going to get everything setup on docker, then do a test deploy on google cloud.
      • 2016-08-11 22402, 2016

      • ruaok
        once that works on some level, I'll go order a pile of machines of hetzner.
      • 2016-08-11 22436, 2016

      • Kode_
        interesting, why docker?
      • 2016-08-11 22405, 2016

      • ruaok
        we started using chef, but that was write once debug many.
      • 2016-08-11 22416, 2016

      • ruaok
        docker has been considerably simpler for our needs.
      • 2016-08-11 22442, 2016

      • Kode_
        well.. they aren't really comparable are they? chef automates setup whereas docker is a container
      • 2016-08-11 22445, 2016

      • Kode_
        so the alternatives to docker would be lxc or kvm, whereas the alternatives to chef would be ansible or puppet
      • 2016-08-11 22411, 2016

      • ruaok
        we've looked at ansible, puppet and chef. they all suck.
      • 2016-08-11 22427, 2016

      • Kode_
        I don't disagree :)
      • 2016-08-11 22428, 2016

      • ruaok
        and our configuration needs are simple enough that we can do it with docker.
      • 2016-08-11 22412, 2016

      • Kode_
        don't get me wrong, I'm a docker advocate as part of the linuxserver.io docker group, but it's not as mature as say lxc so was just a bit surprised
      • 2016-08-11 22414, 2016

      • ruaok
        I agree that it is an apples and oranges comparison, but the isolation aspects of docker make writing configurations for our setup much easier.
      • 2016-08-11 22435, 2016

      • Kode_ is now known as KodeStar
      • 2016-08-11 22458, 2016

      • ruaok
        that is what zas got excited about. Also a long time MB contributor works for Docker. :)
      • 2016-08-11 22411, 2016

      • ruaok
        we all started playing with it and really liked it.
      • 2016-08-11 22444, 2016

      • ruaok
        ok, next question: why the fuck did bookbrainz not start.
      • 2016-08-11 22432, 2016

      • ruaok
        what is the app server for bookbrainz? anyone know?
      • 2016-08-11 22443, 2016

      • KodeStar
        fair enough, it's a good platform, and my group is responsible for the most popular musicbrainz docker ;)
      • 2016-08-11 22455, 2016

      • KodeStar
      • 2016-08-11 22410, 2016

      • ruaok
        ah, I've been using musicbrainz-docker from jsturgis.
      • 2016-08-11 22456, 2016

      • KodeStar
        I can't even see that one on the docker hub? lol
      • 2016-08-11 22400, 2016

      • drsaunders joined the channel
      • 2016-08-11 22424, 2016

      • ruaok
      • 2016-08-11 22434, 2016

      • ariscop joined the channel
      • 2016-08-11 22438, 2016

      • ruaok
        Leftmost, LordSputnik: why does bookbrainz on sakura not start?
      • 2016-08-11 22459, 2016

      • KodeStar
        ahh, it's not on the hub just on github?
      • 2016-08-11 22407, 2016

      • mihaitish joined the channel
      • 2016-08-11 22433, 2016

      • ruaok
        we've had so many start on reboot issues, I'm just going to leave bookbrainz.org down until someone decides to get more serious about it.
      • 2016-08-11 22442, 2016

      • ruaok declares an SEP field
      • 2016-08-11 22415, 2016

      • adhawkins
        ruaok: Will do.
      • 2016-08-11 22421, 2016

      • ruaok
        thanks!
      • 2016-08-11 22421, 2016

      • KodeStar
        doh, I'm actually interested in bookbrainz for a future possible project, lol
      • 2016-08-11 22410, 2016

      • ruaok
      • 2016-08-11 22411, 2016

      • ruaok
        :)
      • 2016-08-11 22416, 2016

      • ruaok goes to be evil
      • 2016-08-11 22419, 2016

      • pingupingu has quit
      • 2016-08-11 22434, 2016

      • ruaok
        LordSputnik: look, test.bookbrainz.org is down too
      • 2016-08-11 22414, 2016

      • ruaok
        Gentlecat: any idea why metabrainz.org cannot connect to postres after a reboot?
      • 2016-08-11 22431, 2016

      • ruaok
        the replication packets are not working.
      • 2016-08-11 22434, 2016

      • Gentlecat
        let me read the backlog
      • 2016-08-11 22459, 2016

      • ruaok
        I rebooted sakura and tons of shit broke, so I am slowly working my way through all the issues, hoping to fix them.
      • 2016-08-11 22401, 2016

      • Gentlecat
        CB seems to be connecting fine, mbspotify too
      • 2016-08-11 22412, 2016

      • ruaok
        meb for other pages too.
      • 2016-08-11 22417, 2016

      • ruaok
        which is really weird.
      • 2016-08-11 22432, 2016

      • Gentlecat
        but not for replication?
      • 2016-08-11 22437, 2016

      • ruaok
        correct.