#metabrainz

/

      • Nyanko-sensei joined the channel
      • 2020-12-01 33613, 2020

      • CatQuest has quit
      • 2020-12-01 33643, 2020

      • CatQuest joined the channel
      • 2020-12-01 33643, 2020

      • CatQuest has quit
      • 2020-12-01 33643, 2020

      • CatQuest joined the channel
      • 2020-12-01 33631, 2020

      • sumedh joined the channel
      • 2020-12-01 33625, 2020

      • thomasross has quit
      • 2020-12-01 33636, 2020

      • sumedh has quit
      • 2020-12-01 33607, 2020

      • sumedh joined the channel
      • 2020-12-01 33604, 2020

      • sumedh has quit
      • 2020-12-01 33603, 2020

      • Nyanko-sensei has quit
      • 2020-12-01 33640, 2020

      • Nyanko-sensei joined the channel
      • 2020-12-01 33603, 2020

      • d4rkie joined the channel
      • 2020-12-01 33616, 2020

      • Nyanko-sensei has quit
      • 2020-12-01 33650, 2020

      • sampsyo has quit
      • 2020-12-01 33638, 2020

      • sampsyo joined the channel
      • 2020-12-01 33642, 2020

      • MajorLurker has quit
      • 2020-12-01 33652, 2020

      • MajorLurker joined the channel
      • 2020-12-01 33602, 2020

      • sumedh joined the channel
      • 2020-12-01 33601, 2020

      • Gazooo79494 has quit
      • 2020-12-01 33631, 2020

      • Nyanko-sensei joined the channel
      • 2020-12-01 33645, 2020

      • Gazooo79494 joined the channel
      • 2020-12-01 33625, 2020

      • d4rkie has quit
      • 2020-12-01 33614, 2020

      • ruaok
        alastairp: which copy of bono-data-set hoster is live on bono right now? your new PR branch? I have one tiny update to make (table name change for year lookup).
      • 2020-12-01 33623, 2020

      • ruaok
        and then sentry fixes too. :)
      • 2020-12-01 33625, 2020

      • alastairp
        no, I didn't release anything
      • 2020-12-01 33632, 2020

      • ruaok
        thanks for that!
      • 2020-12-01 33646, 2020

      • ruaok
        ah, those sentry alerts are from testing, then?
      • 2020-12-01 33650, 2020

      • alastairp
        correct
      • 2020-12-01 33653, 2020

      • ruaok
        ok, kewl.
      • 2020-12-01 33607, 2020

      • ruaok
        let me just commit the table name change to main and deploy an update.
      • 2020-12-01 33624, 2020

      • alastairp
        btw, I'm putting LB unit and int tests on jenkins, so that they don't take hours to run
      • 2020-12-01 33632, 2020

      • ruaok
        +10000
      • 2020-12-01 33609, 2020

      • alastairp
        then I'll get a PR for bono-datasets and LB to use the new version of datasethoster+sentry
      • 2020-12-01 33611, 2020

      • ruaok
        after I pay people, I'll jump back into troi and then playlist serializing... I had some questions for you...
      • 2020-12-01 33659, 2020

      • ruaok
        acrp-year-lookup moved to new mbid-mapping PR table name
      • 2020-12-01 33601, 2020

      • alastairp
        uh oh... questions
      • 2020-12-01 33650, 2020

      • alastairp
        yvanzo: hi, were we talking some months ago about removing test images from williams after running jenkins jobs?
      • 2020-12-01 33655, 2020

      • Mr_Monkey
        MFW I realise it's now December.
      • 2020-12-01 33613, 2020

      • alastairp
        Mr_Monkey: what do you mean? It's still March
      • 2020-12-01 33623, 2020

      • Mr_Monkey
        Right?
      • 2020-12-01 33635, 2020

      • alastairp
        we're going to wake up any time now
      • 2020-12-01 33623, 2020

      • reosarevok
        Mr_Monkey: but that means your holidays are close!
      • 2020-12-01 33635, 2020

      • reosarevok
        Wait, we don't have official holidays. Whoops.
      • 2020-12-01 33651, 2020

      • reosarevok
        You can meet with family and have a lovely time!
      • 2020-12-01 33659, 2020

      • reosarevok
        Wait, COVID restrictions. Whoops.
      • 2020-12-01 33607, 2020

      • reosarevok
        Ok, yeah, why is it December already
      • 2020-12-01 33622, 2020

      • reosarevok
        Well, less time until a vaccine I guess, hopefully. *that* is a positive, yes?
      • 2020-12-01 33650, 2020

      • zas
        alerts due to critical upgrades, unexpected :(
      • 2020-12-01 33604, 2020

      • _45054 joined the channel
      • 2020-12-01 33639, 2020

      • reosarevok
        Ok so the 502s are on your radar
      • 2020-12-01 33643, 2020

      • ruaok
        ugh. need help zas?
      • 2020-12-01 33651, 2020

      • reosarevok waits a bit unless he can be of use
      • 2020-12-01 33603, 2020

      • zas
        not really yet, but I expect mess in our services
      • 2020-12-01 33612, 2020

      • reosarevok
        MB is already 502ing
      • 2020-12-01 33629, 2020

      • Sophist_UK
        Ah - glad that you are already aware of the 502s - I was just going to mention them.
      • 2020-12-01 33631, 2020

      • reosarevok
        Do you expect this to last? Should I tweet?
      • 2020-12-01 33633, 2020

      • ruaok
        that has indeed arrive, zas.
      • 2020-12-01 33649, 2020

      • zas
        a package upgrade shouldn't have restarted services, but it did :(
      • 2020-12-01 33657, 2020

      • ruaok
        oy.
      • 2020-12-01 33606, 2020

      • BrainzGit has quit
      • 2020-12-01 33609, 2020

      • zas
        let's see what recovers or not
      • 2020-12-01 33628, 2020

      • ruaok
        nothing yet. :(
      • 2020-12-01 33633, 2020

      • ruaok
        should things be recovering now?
      • 2020-12-01 33656, 2020

      • zas
        I guess so, but ...
      • 2020-12-01 33606, 2020

      • ruaok
        smells like postgres.
      • 2020-12-01 33613, 2020

      • ruaok
      • 2020-12-01 33624, 2020

      • ruaok
        zas: check pg bouncer.
      • 2020-12-01 33627, 2020

      • ruaok
        did it come back ok?
      • 2020-12-01 33629, 2020

      • Mr_Monkey
        Yeah, just visiting it now
      • 2020-12-01 33633, 2020

      • ruaok
        perhaps restart it?
      • 2020-12-01 33635, 2020

      • Mr_Monkey
        "Unable to acquire a connection"
      • 2020-12-01 33644, 2020

      • Mr_Monkey
        Is everyhing else back up?
      • 2020-12-01 33651, 2020

      • ruaok
        Mr_Monkey: yes, you're providing a good clue that none of our other sites are.
      • 2020-12-01 33659, 2020

      • ruaok
        Mr_Monkey: not yet.
      • 2020-12-01 33603, 2020

      • Mr_Monkey
        OK
      • 2020-12-01 33655, 2020

      • zas
        wait a bit, it seems things are coming back after the storm, but we will find corpses....
      • 2020-12-01 33606, 2020

      • ruaok
        LB won't come back up.
      • 2020-12-01 33615, 2020

      • reosarevok
        Yeah, my local ssh tunnel to pink works but there's no DB connection
      • 2020-12-01 33615, 2020

      • Mr_Monkey
        acousticbrainz.org is unaffected (or already back up)
      • 2020-12-01 33643, 2020

      • ruaok
        zas: how is pg bouncer?
      • 2020-12-01 33645, 2020

      • reosarevok
      • 2020-12-01 33600, 2020

      • reosarevok
        So postgres-pink is running, but psql fails
      • 2020-12-01 33601, 2020

      • Mr_Monkey
        Mmm, spoke too fast. Landing page *seems* unaffected only
      • 2020-12-01 33602, 2020

      • ruaok
        yep, can you please restart pg_bouncer, zas?
      • 2020-12-01 33614, 2020

      • BrainzGit joined the channel
      • 2020-12-01 33625, 2020

      • ruaok
        confirmed.
      • 2020-12-01 33628, 2020

      • ruaok
        DB is not up.
      • 2020-12-01 33654, 2020

      • ruaok
        or rather DB is unreachable. pg bouncer likely.
      • 2020-12-01 33611, 2020

      • BrainzGit
        [bookbrainz-site] MonkeyDo merged pull request #538 (master…typescript): Typescript - Stage 2 https://github.com/bookbrainz/bookbrainz-site/pul…
      • 2020-12-01 33627, 2020

      • zas
        pgbouncer doesn't start
      • 2020-12-01 33636, 2020

      • zas
        wtf
      • 2020-12-01 33639, 2020

      • ruaok
        which machine?
      • 2020-12-01 33628, 2020

      • ruaok
        pink
      • 2020-12-01 33619, 2020

      • ruaok
        zas: postgres on pink doesn't seem to have started correctly.
      • 2020-12-01 33614, 2020

      • zas
        on floyd
      • 2020-12-01 33618, 2020

      • zas
        floyd is main db
      • 2020-12-01 33626, 2020

      • ruaok
        ah
      • 2020-12-01 33628, 2020

      • zas
        I don't get why it doesn't start
      • 2020-12-01 33659, 2020

      • reosarevok
        I'll tweet for now just in case
      • 2020-12-01 33605, 2020

      • zas
        yes please
      • 2020-12-01 33643, 2020

      • Mr_Monkey
        Same
      • 2020-12-01 33640, 2020

      • ruaok
        zas: postgres on floyd is NOT up.
      • 2020-12-01 33650, 2020

      • ruaok
        pg bouncer cannot connect because the db is donw.
      • 2020-12-01 33652, 2020

      • reosarevok
        "Sorry about the 502 errors on MusicBrainz right now! Our database didn't come back up as expected after an upgrade (that we weren't expecting would take it down in the first place). We're working on it! - reo"
      • 2020-12-01 33654, 2020

      • zas
        nope, and I don't know why
      • 2020-12-01 33658, 2020

      • reosarevok
        Sounds about right?
      • 2020-12-01 33610, 2020

      • zas
        reosarevok: yup
      • 2020-12-01 33621, 2020

      • ruaok
        zas: let me try and restart it.
      • 2020-12-01 33622, 2020

      • reosarevok
        Sent
      • 2020-12-01 33628, 2020

      • zas
        ruaok: ok
      • 2020-12-01 33649, 2020

      • ruaok
        yerp, not starting.
      • 2020-12-01 33651, 2020

      • ruaok
        fun,
      • 2020-12-01 33620, 2020

      • zas
        ruaok: let's try a full reboot in case of
      • 2020-12-01 33603, 2020

      • ruaok
        think that will change anything?
      • 2020-12-01 33606, 2020

      • ruaok
        we can try....
      • 2020-12-01 33635, 2020

      • ruaok
        its not evening trying to start pg. wtaf?
      • 2020-12-01 33636, 2020

      • zas
        well, yd cron[24]: (CRON) INFO (Skipping @reboot jobs -- not system startup)
      • 2020-12-01 33645, 2020

      • ruaok
        ok, go for it zas
      • 2020-12-01 33600, 2020

      • ruaok
        I'm not hopeful.
      • 2020-12-01 33621, 2020

      • zas
        can you ping bitmap? we may need him
      • 2020-12-01 33658, 2020

      • zas
        what a mess
      • 2020-12-01 33619, 2020

      • ruaok
        pinged.
      • 2020-12-01 33640, 2020

      • ruaok
        does it normally take this long to reboot floyd?
      • 2020-12-01 33627, 2020

      • ruaok
        this is not promising at all. :(
      • 2020-12-01 33654, 2020

      • zas
        it took time
      • 2020-12-01 33659, 2020

      • zas
        but it's back
      • 2020-12-01 33602, 2020

      • ruaok
        ok, not a reboot issue.
      • 2020-12-01 33610, 2020

      • zas
        though pg container still doesn't start anything
      • 2020-12-01 33616, 2020

      • zas
        yvanzo: around?
      • 2020-12-01 33653, 2020

      • ruaok
        zas: is this repo our DB setup? https://github.com/metabrainz/docker-postgres
      • 2020-12-01 33647, 2020

      • BrainzGit
        [listenbrainz-server] alastair opened pull request #1185 (master…jenkins-tests): Add run script for jenkins integration tests, fix jenkins unit tests https://github.com/metabrainz/listenbrainz-server…
      • 2020-12-01 33612, 2020

      • zas
        ruaok: likely, but image is postgres-floyd
      • 2020-12-01 33618, 2020

      • ruaok
        k
      • 2020-12-01 33636, 2020

      • ruaok
        no dockerfile
      • 2020-12-01 33630, 2020

      • davic joined the channel
      • 2020-12-01 33650, 2020

      • zas
      • 2020-12-01 33607, 2020

      • zas
      • 2020-12-01 33643, 2020

      • Mr_Monkey
        ping me if there's anything I can help with
      • 2020-12-01 33626, 2020

      • zas
        ruaok: we can try to rebuild the container, but that's quite random
      • 2020-12-01 33644, 2020

      • zas
        yvanzo, bitmap: any of you around?
      • 2020-12-01 33621, 2020

      • zas
        I don't know enough about current pg setup, and I know bitmap did a lot of changes recently
      • 2020-12-01 33652, 2020

      • ruaok
        I'm digging.
      • 2020-12-01 33657, 2020

      • ruaok
        try restarting pg bouncer, zas
      • 2020-12-01 33614, 2020

      • zas
        how?
      • 2020-12-01 33628, 2020

      • ruaok
        docker container.
      • 2020-12-01 33635, 2020

      • ruaok
        postgres floyd is up
      • 2020-12-01 33641, 2020

      • CatQuest
        rarg what is happen
      • 2020-12-01 33607, 2020

      • sumedh has quit
      • 2020-12-01 33633, 2020

      • zas
        ruaok: pgbouncer is in postgres-floyd container