#metabrainz

monkey: I can’t drag more out of my brain this Friday, I had a bit of a play with the genre drop downs. Not perfectly happy but something to look at: https://www.figma.com/file/YRbCOtFHBez8XmMdCKbG...

[musicbrainz-android] 14akshaaatt merged pull request #156 (03master…dependabot/gradle/androidx.compose.material3-material3-1.1.0-alpha02): Bump material3 from 1.1.0-alpha01 to 1.1.0-alpha02 https://github.com/metabrainz/musicbrainz-andro...

[musicbrainz-android] 14akshaaatt merged pull request #155 (03master…dependabot/gradle/compose_version-1.3.1): Bump compose_version from 1.3.0 to 1.3.1 https://github.com/metabrainz/musicbrainz-andro...

[musicbrainz-android] 14akshaaatt merged pull request #152 (03master…dependabot/gradle/androidx.test-runner-1.5.1): Bump runner from 1.5.0 to 1.5.1 https://github.com/metabrainz/musicbrainz-andro...

[musicbrainz-android] 14akshaaatt merged pull request #153 (03master…dependabot/gradle/androidx.navigation-navigation-compose-2.6.0-alpha04): Bump navigation-compose from 2.6.0-alpha03 to 2.6.0-alpha04 https://github.com/metabrainz/musicbrainz-andro...

[musicbrainz-android] 14akshaaatt merged pull request #148 (03master…dependabot/gradle/androidx.test.espresso-espresso-intents-3.5.0): Bump espresso-intents from 3.4.0 to 3.5.0 https://github.com/metabrainz/musicbrainz-andro...

[musicbrainz-android] 14akshaaatt merged pull request #150 (03master…dependabot/gradle/androidx.test.ext-junit-1.1.4): Bump junit from 1.1.3 to 1.1.4 https://github.com/metabrainz/musicbrainz-andro...

[musicbrainz-android] 14akshaaatt merged pull request #154 (03master…dependabot/gradle/androidx.compose.material3-material3-window-size-class-1.1.0-alpha02): Bump material3-window-size-class from 1.1.0-alpha01 to 1.1.0-alpha02 https://github.com/metabrainz/musicbrainz-andro...

atj: we get warnings about listen drops on rudi/rex since yesterday, likely related to issues we have

atj: on rex, I see you set up keydb container to listen on 10.2.2.60 only

haproxy tries to connect to 10.2.3.30:13062

zas: I've changed it back now, it was just to see if binding to a specific IP made any difference

see output of : curl -s http://10.2.2.30:8500/v1/health/checks/keydb-ga... -m json.tool

rudi service check is failing, and rex's one doesn't even appear

we need to understand why

I suggest we start over, first we stop rex container, and try to get an healthy status on rudi's instance

with the same version as we have on kiki/herb and same config, it answers from inside the container, but not from outside

so that's something with firewall and/or docker, we need to be able to query keydb from rudi itself before anything

redis-cli hangs https://www.irccloud.com/pastebin/aQIbbB8U/

it seems to indicate keydb cannot answer, so that the traffic coming out is dropped

I'll stop the container on rex

we can tcpdump and check what passes or not

traffic from int to int interface should be implicitly allowed by shorewall (can you check it is?)

but here we have a docker to int traffic

when you stop the instance rex, the error message on rudi changes from timeout to connection refused

btw, making keydb container listening only on 10.2.2.30 seems ok to me, as it should only be used by haproxy and consul health checks (which should both use this IP to connect)

atj: yes, ignore for now, let's focus on rudi instance

it has to pass health check and let us connect from the host itself

I suggest we restrict listening to 10.2.2.30:13062, can you make changes for that on rudi?

OK, sure. I'm in bed at the moment, need to get up and have breakfast before I get into this ;)

hmmm, https://docs.keydb.dev/docs/cluster-tutorial/#k... -> but it works on kiki/herb...

oh sorry, that's early

it suggests to use host mode, but AFAIK we don't on herb & kiki and it works. Also https://hub.docker.com/r/eqalpha/keydb examples don't use host mode

Interesting numbers from this Github annual report -> https://octoverse.github.com/

sigh, bloody fibre issues are driving me mad

setting rudi container to listen on 10.2.2.30

😕

but redis-cli still cannot connect

shorewall policy is fw -> ALL - ACCEPT, internal -> fw - ACCEPT, docker -> all - ACCEPT

redis-cli is fw -> docker

also we need to change haproxy to use 10.2.2.x addresses (it seems to use 10.2.3.x)

yes, keydb works (from container, or on docker IP)

tcp check from 10.2.2.x (consul) works too

but connection from localhost using redis-cli doesn't

so the TCP handshake is working

shouldn't docker-proxy be runnning?

atj: dunno, that's internal docker stuff, can you start a test container as you said (running a dummy service)? currently we only have keydb as container in non-host mode

nc in a test container doesn't work

so that rules out keydb as being the source of the issue, docker + firewall, we are missing something. With the nc container, can you connect with source IP being on 10.2.2.x network (using -s option)?

sorry, connection dropped and killed all my sessions

zas: internet connection keeps dropping, hopefully it's because they're doing some diagnostics to investigate my fault

getting great value for money from my 500Mbps connection https://usercontent.irccloud-cdn.com/file/L9kbd...

bitmap: I noticed the weird stuff in https://tickets.metabrainz.org/browse/MBS-12705 when testing your latest PR - not a blocking issue, but a few choices seem odd to me, so maybe we can talk about them (possibly with aerozol too)

MBS-12705: Beta: Make UI for repeatable attributes less confusing

atj: if packets are dropped, they should appear in iptables counters right? which rules have >0 drops?

userland-proxy is set to false in docker daemon.json

https://github.com/metabrainz/metabrainz-ansibl... ?

can I set to true and reload docker? not sure if it will take effect though

you can try

do you remember why you set it to false?

well, it is supposed to not be needed anymore but I guess that's when iptables=true

yes, you need it when iptables is disabled

to be honest host network mode would probably make sense for keydb

reloading docker didn't work, I restarted the container but still no docker-proxy processes

can I try it on rex and reboot?

wait, we need to find a way to avoid the reboot

well, stopping and starting docker should work too

but that would stop all containers

yes, but it is better than rebooting

restart docker then?

[musicbrainz-server] 14reosarevok opened pull request #2725 (03master…more-perl-critic-codelayout): Follow Perl::Critic::Policy::CodeLayout rules https://github.com/metabrainz/musicbrainz-serve...

yup

I was wrong, docker restart didn't restart the containers, but it applied the config changes

works now https://www.irccloud.com/pastebin/RSU0rAR7/

keydb sync from rudi -> rex completed

just going to stop the keydb container on rex and make it listen on 10.2.2.60 only

ok, please make changes to ansible, and switch back to keydb 6.3.1

alright

and let's listen only on 10.2.2.x (and make changes to haproxy for it to work)

OK to restart docker on rudi?

yup

running keydb in host mode would get rid of userland-proxy, so it might be a good idea, but we keep it on same port (13062)

rex+rudi running keydb 6.3.1 and listening on 10.2.2.x only

sync is working in both directions

this is why the TCP handshake was working without userland-proxy enabled https://www.irccloud.com/pastebin/c8JLbKCW/

zas: I've pushed a couple of commits to the PR branch. I've got some work I need to get on with for a few hours, you ok to deploy the haproxy updates?

yup, I'll do after lunch

chinmay: Thanks for the advance notice about the whole "wrapping test side effects with act(…)" thing ! I'm hitting this problem in the react 18 update PR, except there's a million of those warnings… At least I know a bit more about where that's coming from now.

monkey: hahaha I saw that test log!

Makes me want to cry

All the best xD

atj: haproxy changes deployed, redis-cli -h 10.2.3.254 works

yay

one thing: I think we should make rudi's keydb backup on rex, and rex's keydb backup on rudi, currently rudi's backup on both

what do you think?

yeah, that seems sensible

also, I don't think we need to sync with kiki/herb instances: web services on rudi/rex are different anyway

[musicbrainz-server] 14reosarevok opened pull request #2726 (03master…more-perl-critic-valuesandexpressions): Follow Perl::Critic::Policy::ValuesAndExpressions rules https://github.com/metabrainz/musicbrainz-serve...

atj: https://github.com/metabrainz/metabrainz-ansibl...

do you have a better way to achieve this?

not sure if this is "better" but it works https://www.irccloud.com/pastebin/7aU195DO/

not really better

can we take the "other host in gw_servers" as backup?

basically all nodes in gw_servers should be backups but the node itself (at this time), one line for each

hmmm, let me try and come up with something

I tested with rudi's keydb instance stopped, and it still works, it resyncs on restart

I INCR "test" key with redis-cli from both nodes

no issue

yeah I saw that on split brain resync it checks the timestamps and uses the newest value

LB team: check https://stats.metabrainz.org/d/000000079/alerts-lb

zas: looking

monkey: hi! do you use webscrobbler?

Hello! Yes I do

can you point me to some listen that you submitted using it?

Sure thing

is the TS writer getting tripped up on some WS listens, lucifer ?

mayhem: yes

Interesting

specificially because duration = 3682554000

which is too large to store in an integer

hmm, long track.

loooooooooooooooooooooooooooooooooooooooooooong track

i wonder if its submitting ms in duration instead of duration_ms

Sure sounds like that

i'll drop this particular listen but want to make sure whether the issue is in LB or webscobbler

Still looking for a listen, hang on

mayhem: do you think its reasonable to put a check in the api for max duration of ~68 years (max int value)? :)

Would it be safe to move it to duration_ms in that case?

ts writer's y2k38 i guess lol

monkey: yes its most likely a bug but raising an error should probably prompt the submitting software to fix it.

instead of automatically moving duration to duration_ms.

Understood

lucifer: yes, I think so. we might also be a bit more conservative and limit it to 24 hours or so. :)

sure. let me just check MB for the current max recording length lol

hmm that one looks right