#metabrainz

/

      • alastairp
        I also just restarted acousticbrainz postgres on frank, I hadn't restarted it when the previous log limits were added. removed 100gb of logs. This should now be limited to 5gb as per docker config
      • 2021-02-01 03227, 2021

      • _lucifer
        ruaok, yes
      • 2021-02-01 03239, 2021

      • ruaok
        could I please ask you for a PR for that?
      • 2021-02-01 03256, 2021

      • _lucifer
        sure, what should the expiration timeout?
      • 2021-02-01 03207, 2021

      • ruaok
        because the redis keyspace continues to grow. I doubt its the problem we're chasing now, but it will be a problem again.
      • 2021-02-01 03228, 2021

      • ruaok
        _lucifer: 30m, I suppose.
      • 2021-02-01 03243, 2021

      • _lucifer
        👍
      • 2021-02-01 03259, 2021

      • ruaok
        thanks!
      • 2021-02-01 03202, 2021

      • d4rkie has quit
      • 2021-02-01 03257, 2021

      • zas
        looking at logs I noticed weird ufw log entries https://www.irccloud.com/pastebin/krR8hOfs/
      • 2021-02-01 03225, 2021

      • zas
        ofc whole net 10.2.2.0/24 is allowed by ufw
      • 2021-02-01 03241, 2021

      • zas
        it happened during last load peak
      • 2021-02-01 03257, 2021

      • zas
        and 65405 is the port used by mb website container
      • 2021-02-01 03217, 2021

      • zas
        as far I understand, some packets were marked invalid
      • 2021-02-01 03208, 2021

      • zas
        I enabled logging of invalid packets for the next time
      • 2021-02-01 03254, 2021

      • ruaok
        wtf would traffic between the gateway and the backend be marked as bad?
      • 2021-02-01 03203, 2021

      • ruaok
        I agree, that makes very little sense.
      • 2021-02-01 03215, 2021

      • ruaok
        NIC freaking out?
      • 2021-02-01 03223, 2021

      • BrainzGit
        [musicbrainz-server] reosarevok merged pull request #1884 (beta…MBS-11355): Simplify bottom links to avoid translation parameters https://github.com/metabrainz/musicbrainz-server/…
      • 2021-02-01 03241, 2021

      • zas
        not sure yet, it can be related to conn_track, a temporary lack of resources, or dunno, it happens few times a day, but to port 65405
      • 2021-02-01 03246, 2021

      • zas
      • 2021-02-01 03236, 2021

      • zas
        well, it happens on hip, cage & others
      • 2021-02-01 03230, 2021

      • zas
        but on all, only to port 65405
      • 2021-02-01 03259, 2021

      • ruaok
        well, that is the only port that gets serious traffic, no?
      • 2021-02-01 03207, 2021

      • atj
        is .25 the gateway?
      • 2021-02-01 03226, 2021

      • ruaok
        y
      • 2021-02-01 03215, 2021

      • atj
        AFAIU TCP RST with no payload, means SYN sent to closed port or packet sent for an established connection that was closed by the server
      • 2021-02-01 03231, 2021

      • atj
        I imagine it must be the latter
      • 2021-02-01 03235, 2021

      • atj
        Just had a look at a box with ufw enabled and I think the message would be triggered by "ctstate INVALID"
      • 2021-02-01 03211, 2021

      • atj
        If it correlates with load, perhaps it's due to connection time outs?
      • 2021-02-01 03224, 2021

      • ruaok
        zas: ^^
      • 2021-02-01 03238, 2021

      • atj
        That's my 2c FWIW :)
      • 2021-02-01 03227, 2021

      • zas
        atj: yes, what I think too, but it happens on all nodes, and only for this port (which matches MB website containers)
      • 2021-02-01 03209, 2021

      • atj
        So you have a container listening on 65405 which serves HTTP requests?
      • 2021-02-01 03215, 2021

      • zas
      • 2021-02-01 03222, 2021

      • zas
        (we use docker here)
      • 2021-02-01 03230, 2021

      • zas
        atj: yes
      • 2021-02-01 03255, 2021

      • atj
        Does the gateway send probes to check the port is open and responsive?
      • 2021-02-01 03242, 2021

      • zas
        what do you mean by probes? tcp keepalive?
      • 2021-02-01 03206, 2021

      • zas
        this service is one of the slowest
      • 2021-02-01 03243, 2021

      • atj
        Yeah something like that, it depends on the load balancer. Normally you have something set up to monitor the health of nodes so it doesn't send connections to dead ones.
      • 2021-02-01 03258, 2021

      • zas
      • 2021-02-01 03210, 2021

      • atj
        That link seems relevant. Guess it could be running out of space in the conntrack table
      • 2021-02-01 03204, 2021

      • zas
        unlikely, usual values <10k, max 262k
      • 2021-02-01 03211, 2021

      • zas
        I checked
      • 2021-02-01 03227, 2021

      • atj
        Is that on the GW or container?
      • 2021-02-01 03231, 2021

      • zas
        and in this case I would have something logged by kernel
      • 2021-02-01 03256, 2021

      • atj
        Do you see anything in the NGINX logs at the same time?
      • 2021-02-01 03201, 2021

      • zas
        nope
      • 2021-02-01 03211, 2021

      • zas
        btw, that's on node running the container
      • 2021-02-01 03230, 2021

      • zas
        aka host kernel
      • 2021-02-01 03214, 2021

      • atj
        Well, it's the GW sending the RST. It seems to be saying "you sent me a packet for a connection that was closed"
      • 2021-02-01 03202, 2021

      • atj
        I wouldn't worry about it if you're only getting a few instances.
      • 2021-02-01 03220, 2021

      • atj
        Anyway, I'll be quiet now :)
      • 2021-02-01 03216, 2021

      • Nyanko-sensei joined the channel
      • 2021-02-01 03223, 2021

      • BrainzGit
        [critiquebrainz] amCap1712 opened pull request #340 (master…hotfix): Add cache timeout for mbspotify https://github.com/metabrainz/critiquebrainz/pull…
      • 2021-02-01 03243, 2021

      • Nyanko-sensei has quit
      • 2021-02-01 03235, 2021

      • alastairp
        _lucifer: I think I deleted the CB pylint job on jenkins. oops
      • 2021-02-01 03239, 2021

      • alastairp
        will try and restore it
      • 2021-02-01 03224, 2021

      • reosarevok
        Updating beta
      • 2021-02-01 03238, 2021

      • ruaok
        alastairp: is it crucial that we see that test pass? or should I go for the release of mbspotify?
      • 2021-02-01 03258, 2021

      • _lucifer
        alastairp, 👍
      • 2021-02-01 03206, 2021

      • alastairp
        reosarevok: the pylint one? ignore it
      • 2021-02-01 03210, 2021

      • alastairp
        ruaok: ^
      • 2021-02-01 03210, 2021

      • _lucifer
        ruaok: the tests pass, that's the linting job
      • 2021-02-01 03215, 2021

      • ruaok
        ok
      • 2021-02-01 03228, 2021

      • BrainzGit
        [critiquebrainz] mayhem merged pull request #340 (master…hotfix): Add cache timeout for mbspotify https://github.com/metabrainz/critiquebrainz/pull…
      • 2021-02-01 03231, 2021

      • alastairp
        good time to test backup restoring!
      • 2021-02-01 03201, 2021

      • ruaok goes to look how to deploy mbspotify
      • 2021-02-01 03251, 2021

      • alastairp
        ruaok: this PR is fully within CB, unless you're wanting to deploy mbspotify for other reasons
      • 2021-02-01 03201, 2021

      • _lucifer
        probaly shouldn't be needed ruaok
      • 2021-02-01 03227, 2021

      • ruaok
        oh. so just deploy the critiquebrainz-prod container and that's it?
      • 2021-02-01 03231, 2021

      • alastairp
        yes
      • 2021-02-01 03242, 2021

      • ruaok
        puurfect, thanks you two.
      • 2021-02-01 03251, 2021

      • alastairp
        it's a cache around an api call to mbspotify. completely within CB
      • 2021-02-01 03243, 2021

      • CatQuest
        rrrr
      • 2021-02-01 03200, 2021

      • ruaok
        mbspotify does run in a different container though.
      • 2021-02-01 03253, 2021

      • alastairp
        ruaok: yes, the communication is over an HTTP API
      • 2021-02-01 03209, 2021

      • alastairp
        ruaok: I recall that maybe you and I had a high-level discussion about the future of mbspotify at some time
      • 2021-02-01 03230, 2021

      • ruaok
        it needs to die once I get back to mapping work.
      • 2021-02-01 03235, 2021

      • alastairp
        there it is
      • 2021-02-01 03204, 2021

      • Mr_Monkey
        ruaok: LB test web service fails to stat with `Error starting userland proxy: listen tcp 0.0.0.0:13047: bind: address already in use.`. Weird that the test.lb port would be used without any test_web service running. I `lsof`d the port but I don't know what to do about it other than kill the process. Is that the usual course of action?
      • 2021-02-01 03214, 2021

      • Mr_Monkey
      • 2021-02-01 03241, 2021

      • ruaok
        let me finish this deploy and I'll look.
      • 2021-02-01 03246, 2021

      • Mr_Monkey
        Sure.
      • 2021-02-01 03255, 2021

      • ruaok
        mbspotify deployed.
      • 2021-02-01 03246, 2021

      • ruaok
        Mr_Monkey: you are running start_services and it generates that error?
      • 2021-02-01 03254, 2021

      • Mr_Monkey
        Yes
      • 2021-02-01 03204, 2021

      • alastairp
        are you sure that it's not already running?
      • 2021-02-01 03210, 2021

      • Mr_Monkey
        I also see it when i `docker inspect`the container that gets created (but not started)
      • 2021-02-01 03229, 2021

      • Mr_Monkey
        Certain, I also removed the container
      • 2021-02-01 03244, 2021

      • ruaok
        thats odd
      • 2021-02-01 03249, 2021

      • shivam-kapila
        test.lb.org returns 502 for me
      • 2021-02-01 03259, 2021

      • Mr_Monkey
        none of the running containers list that port either
      • 2021-02-01 03225, 2021

      • Freso
        <BANG>
      • 2021-02-01 03225, 2021

      • Freso
        It’s World Aspergillosis Monday!
      • 2021-02-01 03225, 2021

      • Freso
      • 2021-02-01 03225, 2021

      • Freso
        I haven’t received any mailed in reviews, so…
      • 2021-02-01 03225, 2021

      • Freso
        yvanzo: Go!
      • 2021-02-01 03228, 2021

      • yvanzo
        Hi!
      • 2021-02-01 03238, 2021

      • CatQuest
        .. or imbolc, that too
      • 2021-02-01 03252, 2021

      • yvanzo
        Last week I reviewed pull requests and updated beta.mb.o.
      • 2021-02-01 03257, 2021

      • ruaok
        Freso: the gsoc topic needs to be postponed again. thanks to our DDoS fustercluck
      • 2021-02-01 03207, 2021

      • yvanzo
        (There are only 46 still open pull requests, and 8 drafts.)
      • 2021-02-01 03227, 2021

      • yvanzo
        Next server release is expected soon after this meeting.
      • 2021-02-01 03245, 2021

      • yvanzo
        Besides, I resolved small issues in both MBS and SEARCH.
      • 2021-02-01 03254, 2021

      • Freso
        (Other people up for reviews: Freso, ruaok, alastairp, reosarevok, zas, Mr_Monkey, bitmap, _lucifer, CatQuest, shivam-kapila, diru1100 – anyone else who want to give review, let me know ASAP!)
      • 2021-02-01 03208, 2021

      • yvanzo
        Also helped a bit with investigations on Blackout Thursday.
      • 2021-02-01 03210, 2021

      • reosarevok
        !m yvanzo
      • 2021-02-01 03210, 2021

      • BrainzBot
        You're doing good work, yvanzo!
      • 2021-02-01 03219, 2021

      • yvanzo
        Fin, go reosarevok!
      • 2021-02-01 03223, 2021

      • reosarevok
        Hi!
      • 2021-02-01 03227, 2021

      • ruaok
        blackout thursday, lol
      • 2021-02-01 03257, 2021

      • reosarevok
        Last week I spent time on a lot of smaller tasks
      • 2021-02-01 03242, 2021

      • reosarevok
        Nothing I can think of as super special, but lots of making small changes to PRs for merging, testing stuff etc
      • 2021-02-01 03252, 2021

      • reosarevok
        Also finally finished dropping dodgy decoda links
      • 2021-02-01 03204, 2021

      • CatQuest
        say that 3 times fast!
      • 2021-02-01 03254, 2021

      • CatQuest
        uhm. is everybody stuck now?
      • 2021-02-01 03255, 2021

      • TOPIC: MetaBrainz Community and Development channel | MusicBrainz non-development: #musicbrainz | Channel is logged; see https://musicbrainz.org/doc/IRC for details | Agenda: Reviews [postponed:, GSoC (ruaok)]
      • 2021-02-01 03201, 2021

      • reosarevok
        * Furthermore finally finished dropping dodgy decoda ... dlinks :p
      • 2021-02-01 03205, 2021

      • reosarevok
        Anyway
      • 2021-02-01 03220, 2021

      • Freso
        dlinks? 🤔
      • 2021-02-01 03229, 2021

      • reosarevok
        Don't ask
      • 2021-02-01 03230, 2021

      • CatQuest
        decoda'd links
      • 2021-02-01 03232, 2021

      • reosarevok
        I'm running other MBBE stuff now, and trying to fix a small issue CatQuest just found
      • 2021-02-01 03238, 2021

      • reosarevok
        Before we release the beta code
      • 2021-02-01 03251, 2021

      • reosarevok
        Not sure yet what to do next week, but I'm sure I'll find stuff.
      • 2021-02-01 03257, 2021

      • reosarevok
        CatQuest: what about you?
      • 2021-02-01 03257, 2021

      • CatQuest
        ohohoh
      • 2021-02-01 03200, 2021

      • CatQuest rubs hands
      • 2021-02-01 03210, 2021

      • CatQuest
        (re what reo cna do this week)
      • 2021-02-01 03211, 2021

      • CatQuest
        Meow! `😺`
      • 2021-02-01 03216, 2021

      • CatQuest
        Today is Imbolc, or Clothmas!
      • 2021-02-01 03221, 2021

      • CatQuest
        Last week I worked on a lot of instrument images with reo
      • 2021-02-01 03225, 2021

      • CatQuest
        during that I also do INST-798 updating
      • 2021-02-01 03226, 2021

      • BrainzBot
        INST-798: Review descriptions and change any that are still Wikipedia copy-pastes https://tickets.metabrainz.org/browse/INST-798
      • 2021-02-01 03250, 2021

      • CatQuest
        This week I hope to finally add the jantar! (INST-971) as well as add even more instrument images.
      • 2021-02-01 03250, 2021

      • CatQuest
      • 2021-02-01 03250, 2021

      • CatQuest
        fin, go shivam-kapila
      • 2021-02-01 03251, 2021

      • BrainzBot
      • 2021-02-01 03201, 2021

      • shivam-kapila
        Thanks CatQuest
      • 2021-02-01 03204, 2021

      • CatQuest
        :)
      • 2021-02-01 03206, 2021

      • shivam-kapila
        hi everyone
      • 2021-02-01 03221, 2021

      • shivam-kapila
        last week wasnt much code, busy in exam prep
      • 2021-02-01 03246, 2021

      • shivam-kapila
        but I noted down some potential LB GSoC projects that came to my mind
      • 2021-02-01 03207, 2021

      • shivam-kapila
        Its a very rough draft yet but I will share it for some opinions
      • 2021-02-01 03209, 2021

      • shivam-kapila
      • 2021-02-01 03216, 2021

      • shivam-kapila
        Thanks. fin!
      • 2021-02-01 03225, 2021

      • shivam-kapila
        ruaok: go!
      • 2021-02-01 03232, 2021

      • ruaok
        yo