#metabrainz

/

      • bitmap
        all done, moving on
      • 2017-05-15 13549, 2017

      • bitmap
        new postgres-master container is up, watching the logs to make sure it starts properly
      • 2017-05-15 13542, 2017

      • ruaok
        woooo
      • 2017-05-15 13508, 2017

      • Freso 🙏🏼
      • 2017-05-15 13513, 2017

      • bitmap
        lots of "dial tcp 10.2.2.24:8500: i/o timeout"
      • 2017-05-15 13523, 2017

      • bitmap
        (consul-template)
      • 2017-05-15 13504, 2017

      • bitmap
        exceeded maximum retries... seems like it can't communicate with consul
      • 2017-05-15 13553, 2017

      • zas
        hmmm
      • 2017-05-15 13511, 2017

      • zas
        on which server ?
      • 2017-05-15 13519, 2017

      • bitmap
        bowie
      • 2017-05-15 13521, 2017

      • zas
        10.2.2.24 only?
      • 2017-05-15 13533, 2017

      • bitmap
        well, I get a response outside the container
      • 2017-05-15 13543, 2017

      • bitmap
      • 2017-05-15 13500, 2017

      • zas
        restart dnsmasq container ?
      • 2017-05-15 13516, 2017

      • zas
        done
      • 2017-05-15 13517, 2017

      • zas
        hmmm the container start command misses something ?
      • 2017-05-15 13504, 2017

      • bitmap
        I didn't change anything there...
      • 2017-05-15 13517, 2017

      • zas
        10.2.2.24 can be pinged from the container
      • 2017-05-15 13510, 2017

      • zas
        but no answer from consul
      • 2017-05-15 13531, 2017

      • zas
        2017/05/15 18:22:51 [INFO] agent: (LAN) joined: 3 Err: <nil>
      • 2017-05-15 13534, 2017

      • zas
        ?
      • 2017-05-15 13521, 2017

      • bitmap
        the requests it makes work outside the container though
      • 2017-05-15 13532, 2017

      • ruaok
        last time I had to restart the docker daemon to fix this.
      • 2017-05-15 13533, 2017

      • bitmap
        same ip and port
      • 2017-05-15 13502, 2017

      • zas
        bitmap: i'll restart dockert
      • 2017-05-15 13508, 2017

      • bitmap
        ok
      • 2017-05-15 13530, 2017

      • zas
        same
      • 2017-05-15 13504, 2017

      • zas
        let me check firewal rules
      • 2017-05-15 13523, 2017

      • outsidecontext joined the channel
      • 2017-05-15 13549, 2017

      • zas
        oh, a rule is missing
      • 2017-05-15 13545, 2017

      • zas
        should be fixed
      • 2017-05-15 13549, 2017

      • zas
        bitmap: check
      • 2017-05-15 13556, 2017

      • outsidecontext has quit
      • 2017-05-15 13504, 2017

      • zas
        i wonder why it was working before
      • 2017-05-15 13513, 2017

      • bitmap
        yep, postgres is up
      • 2017-05-15 13532, 2017

      • bitmap
        "database system was not properly shut down; automatic recovery in progress" <- that's concerning, I wonder if docker stop doesn't do the right thing
      • 2017-05-15 13541, 2017

      • bitmap
        it recovered fine though
      • 2017-05-15 13503, 2017

      • ruaok
        I've seen the docker restart not affect containers and at other times kill them violently.
      • 2017-05-15 13520, 2017

      • zas
        docker stop may kill the thing, if too slow, one needs to set up delays
      • 2017-05-15 13530, 2017

      • ruaok
        next is postgres slave?
      • 2017-05-15 13538, 2017

      • bitmap
        yes, moving on
      • 2017-05-15 13508, 2017

      • zas
        also the container has to handle signals properly ... which isn't always trivial
      • 2017-05-15 13525, 2017

      • Mineo joined the channel
      • 2017-05-15 13550, 2017

      • zas
        bitmap: you removed the pg container and build it again right ? because the firewall rule wasn't there since the start, so i think the network mode changed, else i don't see how it could have worked til now
      • 2017-05-15 13525, 2017

      • bitmap
        I removed the container and started a new one
      • 2017-05-15 13538, 2017

      • zas
        yes, commands were different
      • 2017-05-15 13556, 2017

      • bitmap
        they were?
      • 2017-05-15 13509, 2017

      • zas
        else i don't see how it could have worked
      • 2017-05-15 13534, 2017

      • zas
        possibly network=host before ?
      • 2017-05-15 13515, 2017

      • bitmap
        slave is back up
      • 2017-05-15 13517, 2017

      • bitmap
        not sure...
      • 2017-05-15 13525, 2017

      • bitmap
        I'll have to check git history
      • 2017-05-15 13554, 2017

      • bitmap
        I think we can move on now
      • 2017-05-15 13523, 2017

      • bitmap
        next one is for zas :)
      • 2017-05-15 13544, 2017

      • ruaok
        > Bring all sites back up except MusicBrainz (prod, beta), the search server, and the CAA:
      • 2017-05-15 13548, 2017

      • ruaok
        go zas
      • 2017-05-15 13558, 2017

      • zas
        i do
      • 2017-05-15 13557, 2017

      • ruaok
        metabrainz up
      • 2017-05-15 13515, 2017

      • zas
        critiquebrainz upstream is down
      • 2017-05-15 13530, 2017

      • zas
        but gateways are ok
      • 2017-05-15 13552, 2017

      • ruaok
        go on with other bits? the start services later on should bring them back up...
      • 2017-05-15 13554, 2017

      • zas
        i checked, all are ok on nginx side, but some upstream are down
      • 2017-05-15 13552, 2017

      • ruaok
        can we just go on?
      • 2017-05-15 13522, 2017

      • zas
        yes, 2 are missing backends
      • 2017-05-15 13527, 2017

      • zas
        the rest is ok
      • 2017-05-15 13500, 2017

      • ZarkBit has quit
      • 2017-05-15 13524, 2017

      • zas
        upgrades done, i'll reboot when we'll do docker upgrade, nothing critical
      • 2017-05-15 13550, 2017

      • bitmap
        ok, my turn
      • 2017-05-15 13541, 2017

      • bitmap
        containers removed, going to start upgrade.sh
      • 2017-05-15 13557, 2017

      • ruaok
        wooo
      • 2017-05-15 13507, 2017

      • ruaok
        how long do you expect that to run?
      • 2017-05-15 13509, 2017

      • bitmap
        the schema changes should finish fairly instantly, I don't remember how long vaccuuming takes
      • 2017-05-15 13534, 2017

      • bitmap
        not running yet, some missing steps I'm adding
      • 2017-05-15 13539, 2017

      • ruaok
        k
      • 2017-05-15 13522, 2017

      • yvanzo
        ruaok: no fun for me, I have been off for days :(
      • 2017-05-15 13532, 2017

      • bitmap
        ok, fingers crossed
      • 2017-05-15 13517, 2017

      • ruaok
        I hope it is something fun.
      • 2017-05-15 13539, 2017

      • ruaok
        and not being stuck in bed, sick. :(
      • 2017-05-15 13558, 2017

      • yvanzo
        no, visiting ppl who are.
      • 2017-05-15 13527, 2017

      • bitmap
        fixing some shell script error, sigh
      • 2017-05-15 13529, 2017

      • Freso
        yvanzo: :( Family?
      • 2017-05-15 13552, 2017

      • Freso heads off to https://thesession.org/sessions/1499 now, since he will actually be able to make it there for once
      • 2017-05-15 13508, 2017

      • arbenina_ has quit
      • 2017-05-15 13521, 2017

      • zas
        bitmap: tell if you need help on smt
      • 2017-05-15 13525, 2017

      • bitmap
      • 2017-05-15 13555, 2017

      • ruaok
        wut?
      • 2017-05-15 13519, 2017

      • bitmap
        that seems like a really old "most recent" packet
      • 2017-05-15 13529, 2017

      • bitmap
        nov 2016, do we make those anymore?
      • 2017-05-15 13554, 2017

      • ruaok
        not sure. I feel that we can dump them, given our new business model.
      • 2017-05-15 13501, 2017

      • ruaok
        let everyone use hourly.
      • 2017-05-15 13534, 2017

      • ruaok
        if you want to ditch them right now and move on, go for it.
      • 2017-05-15 13551, 2017

      • bitmap
        ok. really confusing what happened there though
      • 2017-05-15 13528, 2017

      • bitmap
        weekly too I guess
      • 2017-05-15 13542, 2017

      • SothoTalKer
        erm um, i am banned? :O
      • 2017-05-15 13553, 2017

      • bitmap
        last one in there is from nov 2016 too
      • 2017-05-15 13517, 2017

      • ruaok
        lol
      • 2017-05-15 13520, 2017

      • reosarevok
        SothoTalKer: yes, probably
      • 2017-05-15 13540, 2017

      • SothoTalKer
        yay, finally my work has put to an hold and i can be lazy
      • 2017-05-15 13505, 2017

      • bitmap
        ok now upgrade.sh is actually running
      • 2017-05-15 13546, 2017

      • ruaok
        k
      • 2017-05-15 13557, 2017

      • bitmap
        vacuuming
      • 2017-05-15 13518, 2017

      • bitmap
        damnit "canceling statement due to statement timeout"
      • 2017-05-15 13543, 2017

      • bitmap
        I'll do the vacuum in a psql shell
      • 2017-05-15 13528, 2017

      • bitmap
        we do disable the timeout for that, but maybe it broke at some point (perhaps when we switched pgbouncer's transaction mode)
      • 2017-05-15 13502, 2017

      • ruaok
        I think we should just connect directly for vacuuming.
      • 2017-05-15 13505, 2017

      • bitmap
        hmm, but it connects directly to postgres.
      • 2017-05-15 13512, 2017

      • bitmap
        yeah, it does. not sure why it did that then
      • 2017-05-15 13527, 2017

      • ruaok
        odd.
      • 2017-05-15 13521, 2017

      • bitmap
        oh, I see. it's only set for queries run through perl. it doesn't update the timeout if you use ./admin/psql
      • 2017-05-15 13524, 2017

      • bitmap
        I'll file a bug for that
      • 2017-05-15 13523, 2017

      • bitmap
        I'll do some of these steps which don't need to wait for the vacuum
      • 2017-05-15 13525, 2017

      • drsaunder joined the channel
      • 2017-05-15 13529, 2017

      • zas
        afk 5 mins
      • 2017-05-15 13539, 2017

      • ruaok finishes dinner
      • 2017-05-15 13542, 2017

      • ruaok
        perfect timing, zas
      • 2017-05-15 13545, 2017

      • SothoTalKer
        the addicts need their fixes =(
      • 2017-05-15 13503, 2017

      • bitmap
        vacuum done
      • 2017-05-15 13519, 2017

      • zas
        back
      • 2017-05-15 13530, 2017

      • bitmap
        sanity checking things
      • 2017-05-15 13552, 2017

      • CatQuest
        in this schema update: bitmap cleans his house!
      • 2017-05-15 13557, 2017

      • CatQuest j/k
      • 2017-05-15 13558, 2017

      • CatQuest
        :D
      • 2017-05-15 13515, 2017

      • ruaok
        what can I help with bitmap ?
      • 2017-05-15 13525, 2017

      • bitmap
        ok, some changes were missing from the compiled schema scripts; I fixed that and applied the changes manually
      • 2017-05-15 13501, 2017

      • bitmap
        the correct scripts are now in master
      • 2017-05-15 13530, 2017

      • bitmap
        dbmirror tables look correct and only have the single replication_control row update
      • 2017-05-15 13544, 2017

      • bitmap
        it looks fine to me
      • 2017-05-15 13539, 2017

      • bitmap
        we can proceed
      • 2017-05-15 13521, 2017

      • ruaok
        great!
      • 2017-05-15 13534, 2017

      • bitmap
        I'm crossing out the docker-server-configs update since I ran it on most nodes already, including kiki
      • 2017-05-15 13513, 2017

      • ruaok
        zas: still here?
      • 2017-05-15 13514, 2017

      • CatQuest
        oioioi
      • 2017-05-15 13518, 2017

      • zas
        yup
      • 2017-05-15 13530, 2017

      • ruaok
        good good.
      • 2017-05-15 13542, 2017

      • ruaok
        zas: can you prepare for "Remove downtime for the following services"?
      • 2017-05-15 13551, 2017

      • zas
        i did already
      • 2017-05-15 13508, 2017

      • zas
        waiting for green light
      • 2017-05-15 13512, 2017

      • CatQuest
        still 503 tho
      • 2017-05-15 13526, 2017

      • ruaok
        great.
      • 2017-05-15 13556, 2017

      • SothoTalKer
        <3
      • 2017-05-15 13548, 2017

      • bitmap
        ruaok: oh, did you push the new search-server images?
      • 2017-05-15 13556, 2017

      • ruaok
        yes
      • 2017-05-15 13503, 2017

      • ruaok
        all verified md5sums
      • 2017-05-15 13521, 2017

      • bitmap
      • 2017-05-15 13529, 2017

      • ruaok
        oh sorry, no.
      • 2017-05-15 13536, 2017

      • ruaok
        I pushed wars/jars.
      • 2017-05-15 13540, 2017

      • ruaok
        not actual search images.
      • 2017-05-15 13546, 2017

      • ruaok
        I can get on that.
      • 2017-05-15 13500, 2017

      • bitmap
        okay, thanks
      • 2017-05-15 13519, 2017

      • ruaok
        builds triggered on docker hub.