#metabrainz

/

      • Lotheric joined the channel
      • 2021-01-27 02745, 2021

      • xuvxuv has quit
      • 2021-01-27 02702, 2021

      • BrainzGit
        [bookbrainz-site] MonkeyDo opened pull request #552 (master…snyk-upgrade-f744969d2d654e17931b5b4d9772de43): [Snyk] Upgrade swagger-jsdoc from 4.0.0 to 4.3.2 https://github.com/bookbrainz/bookbrainz-site/pul…
      • 2021-01-27 02717, 2021

      • d4rkie has quit
      • 2021-01-27 02710, 2021

      • Nyanko-sensei joined the channel
      • 2021-01-27 02748, 2021

      • flamingspinach_ is now known as fs
      • 2021-01-27 02736, 2021

      • d4rkie joined the channel
      • 2021-01-27 02726, 2021

      • D4RK-PH0_ has quit
      • 2021-01-27 02721, 2021

      • travis-ci joined the channel
      • 2021-01-27 02722, 2021

      • travis-ci
        Project bookbrainz-site build #3628: passed in 4 min 17 sec: https://travis-ci.org/bookbrainz/bookbrainz-site/…
      • 2021-01-27 02722, 2021

      • travis-ci has left the channel
      • 2021-01-27 02731, 2021

      • nobodyrocks[m] has quit
      • 2021-01-27 02731, 2021

      • goldenshimmer has quit
      • 2021-01-27 02745, 2021

      • devorizon[m] has quit
      • 2021-01-27 02746, 2021

      • kepstin has quit
      • 2021-01-27 02759, 2021

      • devorizon[m] joined the channel
      • 2021-01-27 02759, 2021

      • kepstin joined the channel
      • 2021-01-27 02727, 2021

      • maxlath[m] has quit
      • 2021-01-27 02735, 2021

      • lorenzuru has quit
      • 2021-01-27 02749, 2021

      • SamThursfield[m] has quit
      • 2021-01-27 02754, 2021

      • devorizon[m] has quit
      • 2021-01-27 02756, 2021

      • kepstin has quit
      • 2021-01-27 02757, 2021

      • reg[m] has quit
      • 2021-01-27 02718, 2021

      • joshuaboniface has quit
      • 2021-01-27 02752, 2021

      • lorenzuru joined the channel
      • 2021-01-27 02736, 2021

      • goldenshimmer joined the channel
      • 2021-01-27 02703, 2021

      • nobodyrocks[m] joined the channel
      • 2021-01-27 02752, 2021

      • joshuaboniface joined the channel
      • 2021-01-27 02732, 2021

      • reg[m] joined the channel
      • 2021-01-27 02739, 2021

      • Nyanko-sensei has quit
      • 2021-01-27 02756, 2021

      • lorenzuru has quit
      • 2021-01-27 02715, 2021

      • Nyanko-sensei joined the channel
      • 2021-01-27 02732, 2021

      • goldenshimmer has quit
      • 2021-01-27 02753, 2021

      • D4RK-PH0_ joined the channel
      • 2021-01-27 02753, 2021

      • nobodyrocks[m] has quit
      • 2021-01-27 02715, 2021

      • Nyanko-sensei has quit
      • 2021-01-27 02751, 2021

      • joshuaboniface has quit
      • 2021-01-27 02728, 2021

      • reg[m] has quit
      • 2021-01-27 02749, 2021

      • maxlath[m] joined the channel
      • 2021-01-27 02711, 2021

      • SamThursfield[m] joined the channel
      • 2021-01-27 02700, 2021

      • kepstin joined the channel
      • 2021-01-27 02743, 2021

      • devorizon[m] joined the channel
      • 2021-01-27 02731, 2021

      • lorenzuru joined the channel
      • 2021-01-27 02715, 2021

      • nobodyrocks[m] joined the channel
      • 2021-01-27 02729, 2021

      • goldenshimmer joined the channel
      • 2021-01-27 02755, 2021

      • joshuaboniface joined the channel
      • 2021-01-27 02736, 2021

      • Darkloke joined the channel
      • 2021-01-27 02701, 2021

      • reg[m] joined the channel
      • 2021-01-27 02727, 2021

      • aereaux has quit
      • 2021-01-27 02735, 2021

      • yvanzo
        Most of main *brainz.org websites are down probably since 4:30 UTC
      • 2021-01-27 02719, 2021

      • yvanzo
        (except tickets, community, stats, office and probably a few other ones.)
      • 2021-01-27 02758, 2021

      • shivam-kapila
        Chatlogs up
      • 2021-01-27 02758, 2021

      • shivam-kapila
        Lb down
      • 2021-01-27 02756, 2021

      • sumedh joined the channel
      • 2021-01-27 02704, 2021

      • reosarevok
        yvanzo: heh, wonder if https://app.circleci.com/pipelines/github/metabra… is caused by that somehow? :/
      • 2021-01-27 02736, 2021

      • yvanzo
        reosarevok: do you mean more donations are needed? ;)
      • 2021-01-27 02757, 2021

      • aereaux joined the channel
      • 2021-01-27 02732, 2021

      • aereaux has quit
      • 2021-01-27 02710, 2021

      • reosarevok
        Absolutely
      • 2021-01-27 02724, 2021

      • reosarevok
        I just don't remember if that code hits MeB for the test
      • 2021-01-27 02707, 2021

      • reosarevok
        Yeah, it does, so that explains that
      • 2021-01-27 02757, 2021

      • yvanzo
        I submitted an issue to Hetzner Cloud (I don’t have access to the console for dedicated servers) with details about hip/metabrainz.org.
      • 2021-01-27 02746, 2021

      • yvanzo
        Got to go now, zas will take over when available.
      • 2021-01-27 02728, 2021

      • sumedh has quit
      • 2021-01-27 02711, 2021

      • sumedh joined the channel
      • 2021-01-27 02747, 2021

      • mglubb joined the channel
      • 2021-01-27 02758, 2021

      • mglubb
        Morning! (well, it is here). Is MusicBrainz's replication API down? I'm getting no response from https://metabrainz.org/api/musicbrainz/replicatio…
      • 2021-01-27 02728, 2021

      • ruaok
        Zas: ping
      • 2021-01-27 02744, 2021

      • zas
        Pong, what's happening?
      • 2021-01-27 02757, 2021

      • ruaok
        Massive 500 errors.
      • 2021-01-27 02735, 2021

      • ruaok
        MB is effectively down.
      • 2021-01-27 02731, 2021

      • mglubb
        :/ I'll leave you to it!
      • 2021-01-27 02733, 2021

      • zas
        hmm
      • 2021-01-27 02716, 2021

      • ruaok
        zas: pre-coffee, but I can try to help. what can I do?
      • 2021-01-27 02721, 2021

      • ruaok waves at mglubb
      • 2021-01-27 02701, 2021

      • zas
        I can login everywhere, and ping all servers. Did you take any action already (apart hetzner ticket)?
      • 2021-01-27 02704, 2021

      • ruaok
        I see more than just MB. everything is down.
      • 2021-01-27 02713, 2021

      • zas
        pre-pre-coffee here too
      • 2021-01-27 02719, 2021

      • ruaok
        no action yet, just sat down at computer.
      • 2021-01-27 02736, 2021

      • ruaok
        seems like a gateway issue to me.
      • 2021-01-27 02759, 2021

      • zas
        switching to herb
      • 2021-01-27 02728, 2021

      • ruaok
        lemmy is receiving a trickle of traffic.
      • 2021-01-27 02747, 2021

      • ruaok
        > Jan 27 08:45:34 lemmy consul-template[28]: ("key(docker-server-configs/LB/config.prod.json/sentry/datasets_dsn)") Consul returned no data (does the path exist?)
      • 2021-01-27 02751, 2021

      • ruaok
        consul problems?
      • 2021-01-27 02731, 2021

      • ruaok
        ziiip traffic!
      • 2021-01-27 02746, 2021

      • ruaok
        kiki's drive must be dying then.
      • 2021-01-27 02718, 2021

      • ruaok
        mglubb: all fixed now. :)
      • 2021-01-27 02734, 2021

      • mglubb
        Many thanks! Sorry for the bumpy wake up call!
      • 2021-01-27 02752, 2021

      • ruaok trundles off to make the juice of life
      • 2021-01-27 02748, 2021

      • sumedh has quit
      • 2021-01-27 02730, 2021

      • diru1100 joined the channel
      • 2021-01-27 02731, 2021

      • diru1100
        Afternoon!!
      • 2021-01-27 02714, 2021

      • BrainzGit has quit
      • 2021-01-27 02735, 2021

      • BrainzGit joined the channel
      • 2021-01-27 02743, 2021

      • ruaok
        zas: what happened now?
      • 2021-01-27 02722, 2021

      • zas
        dunno
      • 2021-01-27 02748, 2021

      • zas
        we have traffic coming in
      • 2021-01-27 02708, 2021

      • ruaok
        check consul.
      • 2021-01-27 02719, 2021

      • zas
        I did, seems ok
      • 2021-01-27 02724, 2021

      • ruaok
        lemmy has no more traffic, but consul error messages a plenty
      • 2021-01-27 02730, 2021

      • ruaok
        Jan 27 09:03:31 lemmy consul-template[28]: ("key(docker-server-configs/LB/config.prod.json/sentry/datasets_dsn)") Consul returned no data (does the path exist?)
      • 2021-01-27 02709, 2021

      • ruaok
        how about sentry, is it dead?
      • 2021-01-27 02717, 2021

      • ruaok
        ("key(docker-server-configs/LB/config.prod.json/sentry/datasets_dsn)") Consul returned no data (does the path exist?)
      • 2021-01-27 02745, 2021

      • zas
        I see weird errors in consul logs
      • 2021-01-27 02756, 2021

      • yef has quit
      • 2021-01-27 02701, 2021

      • ruaok
        mem usage for consul agent went up: https://stats.metabrainz.org/d/000000051/hetzner-…
      • 2021-01-27 02712, 2021

      • ruaok
        might be a sypmtom, not a cause
      • 2021-01-27 02754, 2021

      • yef joined the channel
      • 2021-01-27 02754, 2021

      • yef has quit
      • 2021-01-27 02754, 2021

      • yef joined the channel
      • 2021-01-27 02739, 2021

      • mglubb has quit
      • 2021-01-27 02742, 2021

      • BrainzGit has left the channel
      • 2021-01-27 02701, 2021

      • BrainzGit joined the channel
      • 2021-01-27 02710, 2021

      • zas
        I don't get what's happening
      • 2021-01-27 02724, 2021

      • zas
        services are up, and we have traffic coming in
      • 2021-01-27 02746, 2021

      • zas
        there are hits on mb website
      • 2021-01-27 02749, 2021

      • ruaok
        consul looks suspect.
      • 2021-01-27 02704, 2021

      • ruaok
        `2021/01/27 09:16:46.024121 [WARN] ("key(docker-server-configs/LB/config.prod.json/sentry/datasets_dsn)") Consul returned no data (does the path exist?)
      • 2021-01-27 02704, 2021

      • ruaok
        `
      • 2021-01-27 02740, 2021

      • ruaok
        that is a static consul value -- not a dynamic service. if that key is not being found git2consul or consul itself are having problems.
      • 2021-01-27 02751, 2021

      • ruaok
        yvanzo: you about?
      • 2021-01-27 02715, 2021

      • ruaok
        bitmap: reosarevok: alastairp : if any of you are around, we could use more eyes on this problem.
      • 2021-01-27 02717, 2021

      • alastairp
        hi
      • 2021-01-27 02726, 2021

      • ruaok
        heya.
      • 2021-01-27 02738, 2021

      • ruaok
        all sites are down and we're unsure why.
      • 2021-01-27 02740, 2021

      • alastairp
        I can spend a bit of time looking at this, I have to prepare a class and do another call shortly though
      • 2021-01-27 02755, 2021

      • ruaok
        do you know where sentry is hosted?
      • 2021-01-27 02756, 2021

      • alastairp
        OK, I'm logging in
      • 2021-01-27 02704, 2021

      • ruaok
        not sure if this is sentry or consul.
      • 2021-01-27 02740, 2021

      • zas
        after a kiki's reboot, I can't ping google from it...
      • 2021-01-27 02715, 2021

      • alastairp
      • 2021-01-27 02716, 2021

      • ruaok
        confirmed. very cute.
      • 2021-01-27 02753, 2021

      • alastairp
        I thought we hadn't made a release of LB yet
      • 2021-01-27 02719, 2021

      • ruaok
        alastairp: I doubt this is LB specific. And I doubt it is sentry.
      • 2021-01-27 02720, 2021

      • alastairp
        let me merge it
      • 2021-01-27 02728, 2021

      • ruaok
        looks more like a consul issue.
      • 2021-01-27 02739, 2021

      • ruaok
        LB is serving the requests it gets, it just gets very very few.
      • 2021-01-27 02747, 2021

      • ruaok
        and this error message is all I get.
      • 2021-01-27 02705, 2021

      • zas
        consul *seems* ok, it has all upstreams, openresty is configured properly
      • 2021-01-27 02717, 2021

      • ruaok
        zas: where does sentry live?
      • 2021-01-27 02732, 2021

      • ruaok
        we have no mention of it in docker server configs or in syswiki.
      • 2021-01-27 02746, 2021

      • alastairp
        it might be in serge
      • 2021-01-27 02757, 2021

      • alastairp
        I see a reference to a sentry volume in serge's borg backup file
      • 2021-01-27 02719, 2021

      • ruaok
        yes, I see it.
      • 2021-01-27 02725, 2021

      • ruaok
        let me sanity check it.
      • 2021-01-27 02739, 2021

      • alastairp
        according to sentry.md it looks like it might be configured with docker-compose instead of the startup scripts
      • 2021-01-27 02755, 2021

      • ruaok
        that explains one thing, thanks.
      • 2021-01-27 02727, 2021

      • ruaok
        from the sentry worker log:
      • 2021-01-27 02736, 2021

      • ruaok
      • 2021-01-27 02717, 2021

      • ruaok
        let me restart sentry, just for the hell of it.