#metabrainz

/

      • wargreen_ has quit
      • 2022-11-11 31500, 2022

      • wargreen joined the channel
      • 2022-11-11 31503, 2022

      • aerozol
        monkey: I can’t drag more out of my brain this Friday, I had a bit of a play with the genre drop downs. Not perfectly happy but something to look at: https://www.figma.com/file/YRbCOtFHBez8XmMdCKbGta…
      • 2022-11-11 31501, 2022

      • MRiddickW joined the channel
      • 2022-11-11 31525, 2022

      • MRiddickW has quit
      • 2022-11-11 31554, 2022

      • chancey joined the channel
      • 2022-11-11 31510, 2022

      • BrainzGit
        [musicbrainz-android] 14akshaaatt merged pull request #156 (03master…dependabot/gradle/androidx.compose.material3-material3-1.1.0-alpha02): Bump material3 from 1.1.0-alpha01 to 1.1.0-alpha02 https://github.com/metabrainz/musicbrainz-android…
      • 2022-11-11 31517, 2022

      • BrainzGit
        [musicbrainz-android] 14akshaaatt merged pull request #155 (03master…dependabot/gradle/compose_version-1.3.1): Bump compose_version from 1.3.0 to 1.3.1 https://github.com/metabrainz/musicbrainz-android…
      • 2022-11-11 31526, 2022

      • BrainzGit
        [musicbrainz-android] 14akshaaatt merged pull request #152 (03master…dependabot/gradle/androidx.test-runner-1.5.1): Bump runner from 1.5.0 to 1.5.1 https://github.com/metabrainz/musicbrainz-android…
      • 2022-11-11 31533, 2022

      • BrainzGit
        [musicbrainz-android] 14akshaaatt merged pull request #153 (03master…dependabot/gradle/androidx.navigation-navigation-compose-2.6.0-alpha04): Bump navigation-compose from 2.6.0-alpha03 to 2.6.0-alpha04 https://github.com/metabrainz/musicbrainz-android…
      • 2022-11-11 31500, 2022

      • BrainzGit
        [musicbrainz-android] 14akshaaatt merged pull request #148 (03master…dependabot/gradle/androidx.test.espresso-espresso-intents-3.5.0): Bump espresso-intents from 3.4.0 to 3.5.0 https://github.com/metabrainz/musicbrainz-android…
      • 2022-11-11 31536, 2022

      • BrainzGit
        [musicbrainz-android] 14akshaaatt merged pull request #150 (03master…dependabot/gradle/androidx.test.ext-junit-1.1.4): Bump junit from 1.1.3 to 1.1.4 https://github.com/metabrainz/musicbrainz-android…
      • 2022-11-11 31542, 2022

      • BrainzGit
        [musicbrainz-android] 14akshaaatt merged pull request #154 (03master…dependabot/gradle/androidx.compose.material3-material3-window-size-class-1.1.0-alpha02): Bump material3-window-size-class from 1.1.0-alpha01 to 1.1.0-alpha02 https://github.com/metabrainz/musicbrainz-android…
      • 2022-11-11 31510, 2022

      • chancey has quit
      • 2022-11-11 31549, 2022

      • zas
        atj: we get warnings about listen drops on rudi/rex since yesterday, likely related to issues we have
      • 2022-11-11 31559, 2022

      • zas
      • 2022-11-11 31523, 2022

      • zas
        atj: on rex, I see you set up keydb container to listen on 10.2.2.60 only
      • 2022-11-11 31532, 2022

      • zas
      • 2022-11-11 31546, 2022

      • zas
        haproxy tries to connect to 10.2.3.30:13062
      • 2022-11-11 31521, 2022

      • atj
        zas: I've changed it back now, it was just to see if binding to a specific IP made any difference
      • 2022-11-11 31509, 2022

      • zas
        see output of : curl -s http://10.2.2.30:8500/v1/health/checks/keydb-gate… -m json.tool
      • 2022-11-11 31530, 2022

      • zas
        rudi service check is failing, and rex's one doesn't even appear
      • 2022-11-11 31531, 2022

      • zas
        we need to understand why
      • 2022-11-11 31510, 2022

      • zas
        I suggest we start over, first we stop rex container, and try to get an healthy status on rudi's instance
      • 2022-11-11 31525, 2022

      • zas
        with the same version as we have on kiki/herb and same config, it answers from inside the container, but not from outside
      • 2022-11-11 31533, 2022

      • zas
        so that's something with firewall and/or docker, we need to be able to query keydb from rudi itself before anything
      • 2022-11-11 31517, 2022

      • zas
      • 2022-11-11 31502, 2022

      • zas
        it seems to indicate keydb cannot answer, so that the traffic coming out is dropped
      • 2022-11-11 31506, 2022

      • atj
        I'll stop the container on rex
      • 2022-11-11 31527, 2022

      • zas
        we can tcpdump and check what passes or not
      • 2022-11-11 31527, 2022

      • zas
        traffic from int to int interface should be implicitly allowed by shorewall (can you check it is?)
      • 2022-11-11 31545, 2022

      • zas
        but here we have a docker to int traffic
      • 2022-11-11 31537, 2022

      • atj
        when you stop the instance rex, the error message on rudi changes from timeout to connection refused
      • 2022-11-11 31552, 2022

      • zas
        btw, making keydb container listening only on 10.2.2.30 seems ok to me, as it should only be used by haproxy and consul health checks (which should both use this IP to connect)
      • 2022-11-11 31504, 2022

      • zas
        atj: yes, ignore for now, let's focus on rudi instance
      • 2022-11-11 31523, 2022

      • zas
        it has to pass health check and let us connect from the host itself
      • 2022-11-11 31524, 2022

      • zas
        I suggest we restrict listening to 10.2.2.30:13062, can you make changes for that on rudi?
      • 2022-11-11 31524, 2022

      • atj
        OK, sure. I'm in bed at the moment, need to get up and have breakfast before I get into this ;)
      • 2022-11-11 31540, 2022

      • zas
        hmmm, https://docs.keydb.dev/docs/cluster-tutorial/#key… -> but it works on kiki/herb...
      • 2022-11-11 31554, 2022

      • zas
        oh sorry, that's early
      • 2022-11-11 31504, 2022

      • zas
        it suggests to use host mode, but AFAIK we don't on herb & kiki and it works. Also https://hub.docker.com/r/eqalpha/keydb examples don't use host mode
      • 2022-11-11 31545, 2022

      • SothoTalKer has quit
      • 2022-11-11 31510, 2022

      • zas
        Interesting numbers from this Github annual report -> https://octoverse.github.com/
      • 2022-11-11 31553, 2022

      • SothoTalKer joined the channel
      • 2022-11-11 31549, 2022

      • atj
        sigh, bloody fibre issues are driving me mad
      • 2022-11-11 31552, 2022

      • atj
        setting rudi container to listen on 10.2.2.30
      • 2022-11-11 31501, 2022

      • zas
      • 2022-11-11 31532, 2022

      • atj
        😕
      • 2022-11-11 31538, 2022

      • zas
        but redis-cli still cannot connect
      • 2022-11-11 31505, 2022

      • atj
        shorewall policy is fw -> ALL - ACCEPT, internal -> fw - ACCEPT, docker -> all - ACCEPT
      • 2022-11-11 31534, 2022

      • atj
        redis-cli is fw -> docker
      • 2022-11-11 31517, 2022

      • zas
        also we need to change haproxy to use 10.2.2.x addresses (it seems to use 10.2.3.x)
      • 2022-11-11 31533, 2022

      • atj
      • 2022-11-11 31557, 2022

      • zas
        yes, keydb works (from container, or on docker IP)
      • 2022-11-11 31511, 2022

      • zas
        tcp check from 10.2.2.x (consul) works too
      • 2022-11-11 31528, 2022

      • zas
        but connection from localhost using redis-cli doesn't
      • 2022-11-11 31553, 2022

      • atj
        so the TCP handshake is working
      • 2022-11-11 31518, 2022

      • atj
        shouldn't docker-proxy be runnning?
      • 2022-11-11 31529, 2022

      • zas
        atj: dunno, that's internal docker stuff, can you start a test container as you said (running a dummy service)? currently we only have keydb as container in non-host mode
      • 2022-11-11 31508, 2022

      • atj
        nc in a test container doesn't work
      • 2022-11-11 31546, 2022

      • zas
        so that rules out keydb as being the source of the issue, docker + firewall, we are missing something. With the nc container, can you connect with source IP being on 10.2.2.x network (using -s option)?
      • 2022-11-11 31515, 2022

      • atj
        sorry, connection dropped and killed all my sessions
      • 2022-11-11 31533, 2022

      • atj
        zas: internet connection keeps dropping, hopefully it's because they're doing some diagnostics to investigate my fault
      • 2022-11-11 31513, 2022

      • atj
        getting great value for money from my 500Mbps connection https://usercontent.irccloud-cdn.com/file/L9kbdXl…
      • 2022-11-11 31551, 2022

      • reosarevok
        bitmap: I noticed the weird stuff in https://tickets.metabrainz.org/browse/MBS-12705 when testing your latest PR - not a blocking issue, but a few choices seem odd to me, so maybe we can talk about them (possibly with aerozol too)
      • 2022-11-11 31552, 2022

      • BrainzBot
        MBS-12705: Beta: Make UI for repeatable attributes less confusing
      • 2022-11-11 31502, 2022

      • zas
        atj: if packets are dropped, they should appear in iptables counters right? which rules have >0 drops?
      • 2022-11-11 31520, 2022

      • atj
        zas: OK, I understand why I think
      • 2022-11-11 31546, 2022

      • atj
        userland-proxy is set to false in docker daemon.json
      • 2022-11-11 31523, 2022

      • atj
      • 2022-11-11 31549, 2022

      • atj
        can I set to true and reload docker? not sure if it will take effect though
      • 2022-11-11 31504, 2022

      • zas
        you can try
      • 2022-11-11 31517, 2022

      • atj
        do you remember why you set it to false?
      • 2022-11-11 31533, 2022

      • zas
        well, it is supposed to not be needed anymore but I guess that's when iptables=true
      • 2022-11-11 31505, 2022

      • atj
        yes, you need it when iptables is disabled
      • 2022-11-11 31531, 2022

      • atj
        to be honest host network mode would probably make sense for keydb
      • 2022-11-11 31537, 2022

      • atj
        reloading docker didn't work, I restarted the container but still no docker-proxy processes
      • 2022-11-11 31502, 2022

      • atj
        can I try it on rex and reboot?
      • 2022-11-11 31515, 2022

      • zas
        wait, we need to find a way to avoid the reboot
      • 2022-11-11 31536, 2022

      • atj
        well, stopping and starting docker should work too
      • 2022-11-11 31544, 2022

      • atj
        but that would stop all containers
      • 2022-11-11 31526, 2022

      • zas
        yes, but it is better than rebooting
      • 2022-11-11 31532, 2022

      • atj
        restart docker then?
      • 2022-11-11 31547, 2022

      • BrainzGit
        [musicbrainz-server] 14reosarevok opened pull request #2725 (03master…more-perl-critic-codelayout): Follow Perl::Critic::Policy::CodeLayout rules https://github.com/metabrainz/musicbrainz-server/…
      • 2022-11-11 31533, 2022

      • zas
        yup
      • 2022-11-11 31514, 2022

      • atj
        I was wrong, docker restart didn't restart the containers, but it applied the config changes
      • 2022-11-11 31517, 2022

      • atj
      • 2022-11-11 31558, 2022

      • atj
        keydb sync from rudi -> rex completed
      • 2022-11-11 31518, 2022

      • atj
        just going to stop the keydb container on rex and make it listen on 10.2.2.60 only
      • 2022-11-11 31522, 2022

      • zas
        ok, please make changes to ansible, and switch back to keydb 6.3.1
      • 2022-11-11 31529, 2022

      • atj
        alright
      • 2022-11-11 31552, 2022

      • zas
        and let's listen only on 10.2.2.x (and make changes to haproxy for it to work)
      • 2022-11-11 31530, 2022

      • atj
        OK to restart docker on rudi?
      • 2022-11-11 31545, 2022

      • zas
        yup
      • 2022-11-11 31548, 2022

      • zas
        running keydb in host mode would get rid of userland-proxy, so it might be a good idea, but we keep it on same port (13062)
      • 2022-11-11 31539, 2022

      • atj
        rex+rudi running keydb 6.3.1 and listening on 10.2.2.x only
      • 2022-11-11 31547, 2022

      • atj
        sync is working in both directions
      • 2022-11-11 31515, 2022

      • atj
        this is why the TCP handshake was working without userland-proxy enabled https://www.irccloud.com/pastebin/c8JLbKCW/
      • 2022-11-11 31554, 2022

      • atj
        zas: I've pushed a couple of commits to the PR branch. I've got some work I need to get on with for a few hours, you ok to deploy the haproxy updates?
      • 2022-11-11 31506, 2022

      • zas
        yup, I'll do after lunch
      • 2022-11-11 31501, 2022

      • monkey
        chinmay: Thanks for the advance notice about the whole "wrapping test side effects with act(…)" thing ! I'm hitting this problem in the react 18 update PR, except there's a million of those warnings… At least I know a bit more about where that's coming from now.
      • 2022-11-11 31548, 2022

      • chinmay
        monkey: hahaha I saw that test log!
      • 2022-11-11 31506, 2022

      • monkey
        Makes me want to cry
      • 2022-11-11 31511, 2022

      • chinmay
        All the best xD
      • 2022-11-11 31542, 2022

      • zas
        atj: haproxy changes deployed, redis-cli -h 10.2.3.254 works
      • 2022-11-11 31553, 2022

      • atj
        yay
      • 2022-11-11 31522, 2022

      • zas
        one thing: I think we should make rudi's keydb backup on rex, and rex's keydb backup on rudi, currently rudi's backup on both
      • 2022-11-11 31519, 2022

      • zas
        what do you think?
      • 2022-11-11 31545, 2022

      • atj
        yeah, that seems sensible
      • 2022-11-11 31547, 2022

      • zas
        also, I don't think we need to sync with kiki/herb instances: web services on rudi/rex are different anyway
      • 2022-11-11 31539, 2022

      • BrainzGit
        [musicbrainz-server] 14reosarevok opened pull request #2726 (03master…more-perl-critic-valuesandexpressions): Follow Perl::Critic::Policy::ValuesAndExpressions rules https://github.com/metabrainz/musicbrainz-server/…
      • 2022-11-11 31510, 2022

      • zas
      • 2022-11-11 31506, 2022

      • zas
        do you have a better way to achieve this?
      • 2022-11-11 31521, 2022

      • atj
        not sure if this is "better" but it works https://www.irccloud.com/pastebin/7aU195DO/
      • 2022-11-11 31503, 2022

      • zas
        not really better
      • 2022-11-11 31508, 2022

      • zas
        can we take the "other host in gw_servers" as backup?
      • 2022-11-11 31512, 2022

      • zas
        basically all nodes in gw_servers should be backups but the node itself (at this time), one line for each
      • 2022-11-11 31520, 2022

      • atj
        hmmm, let me try and come up with something
      • 2022-11-11 31554, 2022

      • zas
        I tested with rudi's keydb instance stopped, and it still works, it resyncs on restart
      • 2022-11-11 31521, 2022

      • zas
        I INCR "test" key with redis-cli from both nodes
      • 2022-11-11 31529, 2022

      • zas
        no issue
      • 2022-11-11 31553, 2022

      • atj
        yeah I saw that on split brain resync it checks the timestamps and uses the newest value
      • 2022-11-11 31536, 2022

      • zas
      • 2022-11-11 31555, 2022

      • lucifer
        zas: looking
      • 2022-11-11 31506, 2022

      • lucifer
        monkey: hi! do you use webscrobbler?
      • 2022-11-11 31517, 2022

      • monkey
        Hello! Yes I do
      • 2022-11-11 31528, 2022

      • mayhem thinks this should be interesting
      • 2022-11-11 31536, 2022

      • lucifer
        can you point me to some listen that you submitted using it?
      • 2022-11-11 31543, 2022

      • monkey
        Sure thing
      • 2022-11-11 31505, 2022

      • mayhem
        is the TS writer getting tripped up on some WS listens, lucifer ?
      • 2022-11-11 31516, 2022

      • lucifer
        mayhem: yes
      • 2022-11-11 31522, 2022

      • monkey
        Interesting
      • 2022-11-11 31526, 2022

      • lucifer
        specificially because duration = 3682554000
      • 2022-11-11 31533, 2022

      • lucifer
        which is too large to store in an integer
      • 2022-11-11 31537, 2022

      • mayhem
        hmm, long track.
      • 2022-11-11 31546, 2022

      • mayhem
        loooooooooooooooooooooooooooooooooooooooooooong track
      • 2022-11-11 31506, 2022

      • lucifer
        i wonder if its submitting ms in duration instead of duration_ms
      • 2022-11-11 31553, 2022

      • monkey
        Sure sounds like that
      • 2022-11-11 31557, 2022

      • lucifer
        i'll drop this particular listen but want to make sure whether the issue is in LB or webscobbler
      • 2022-11-11 31524, 2022

      • monkey
        Still looking for a listen, hang on
      • 2022-11-11 31552, 2022

      • lucifer
        mayhem: do you think its reasonable to put a check in the api for max duration of ~68 years (max int value)? :)
      • 2022-11-11 31536, 2022

      • monkey
        Would it be safe to move it to duration_ms in that case?
      • 2022-11-11 31543, 2022

      • lucifer
        ts writer's y2k38 i guess lol
      • 2022-11-11 31513, 2022

      • lucifer
        monkey: yes its most likely a bug but raising an error should probably prompt the submitting software to fix it.
      • 2022-11-11 31527, 2022

      • lucifer
        instead of automatically moving duration to duration_ms.
      • 2022-11-11 31533, 2022

      • monkey
        Understood
      • 2022-11-11 31554, 2022

      • mayhem
        lucifer: yes, I think so. we might also be a bit more conservative and limit it to 24 hours or so. :)
      • 2022-11-11 31533, 2022

      • lucifer
        sure. let me just check MB for the current max recording length lol
      • 2022-11-11 31507, 2022

      • monkey
      • 2022-11-11 31529, 2022

      • lucifer
        hmm that one looks right