#metabrainz

MeB accounts

> <@irc_yvanzo:moviebrainz.org> tweeted about it

* your twitter username?

See https://blog.metabrainz.org/

(links)

all alerts cleared (but load on kiss, soon)

lucifer: might be worth updating docker-server-configs then

yvanzo: it was a temporary move, reverted.

containers are now in sync with what is mentioned in docker-server-configs

👍

everything looks good on my side, alerts are all gone, and stats look normal now

nice, same here

👏

I'll bbiab but can look into deploying MBS-13449 afterward, yvanzo

MBS-13449: Reimplement transactional cache without database locks https://tickets.metabrainz.org/browse/MBS-13449

We finally sorted out this mess. Now back to real work ;)

zas: Where is the script to double-check that there is no longer any duplicate MAC addresses?

ansible -m shell -a 'docker ps --format {% raw %}{{.ID}}{% endraw %} | while read id; do docker inspect $id | jq ".[] .NetworkSettings .Networks .bridge .MacAddress"; done |sort |uniq -c |sort -n |grep -vE "^\s+1|null" || true' docker_servers

I just checked, all clear

👏

lucifer: on kiss & gaga it would be great to prune old containers & images

bitmap: I just did on paco, because of jenkins (it creates images but never clear them, reclaimed 5gb)

lucifer: I mean kiss & wolf (gaga's ok)

bitmap (IRC): just got mail from matthew@kite about a missing repllication packet on meb.org. could you please look into that?

https://metabrainz.org/api/musicbrainz/replicat... is 404

mayhem[m]: it's expected since cron was stopped after packet 165383 from 15:00 UTC. 165384 will be generated in ~10 minutes

fabu, thanks!

zas: done

lucifer: thanks!

I should have timed that better and let cron run at 16:00, will consider that next time

lucifer: What time would work for you on tomorrow?

yvanzo: i should be around most of the day.

Ok, I should be up early but leave early too.

cool, ping me whenever you are around.

👍

bitmap: Do you mean downtime at :15?

åssen gor'e?

åssen går'e?

yvanzo: can you point me to the branch or instructions to follow? I have some time now too and can test it out.

yvanzo: sorry not sure I understand the question

lucifer: https://github.com/metabrainz/musicbrainz-docke...

lucifer: Instructions are the same but using Git Bash

yvanzo: can I use wsl 2 Ubuntu bash instead?

sure if you want

cool

lucifer: Also git should be configured to not mess end of line: See https://docs.google.com/document/d/1bz32duq95jS...

bitmap: scheduling downtime at :15 would let replication packets go next time :)

ah, right

[musicbrainz-server] 14mwiencek merged pull request #3130 (03production…nix-advisory-locks): MBS-13449: Reimplement transactional cache without database locks https://github.com/metabrainz/musicbrainz-serve...

I'm merging production into the beta branch now, is there anything else I should include (or is beta frozen?)

I'm not sure tbh

yvanzo would know

I'm just deploying the cache changes for now

Sure

👍

yvanzo: pg_amqp is failing to build

beta is frozen for a while, only #3130 was missing for a simultaneous beta/prod update

ok, updating beta now

lucifer: is it for `docker-compose build --no-cache db`

yvanzo: `docker-compose build`

i can try building just db too.

but yes this failure was for db image.

lucifer: it's failing here too

probably due to some changes upstream in postgres:12 image

ah :/

they seemed to have moved to bookworm.

not sure if that's new

do you know if we use bookworm or bullseye in production?

for the main cluster?

yes

they use metabrainz/base-image which is based on ubuntu jammy

ah interesting, and install postgres using apt?

yeah from the PGDG repo

I vaguely remember having to patch pg_amqp to add "int" to all of the broker_id arguments, but I don't remember what for.

yup that is what these warnings show as well

do we have a fork of pg_amqp?

nope

this might've been for the postgres-json-dump container, which uses the postgres:16.1-bookworm image

the official repo looks more or less unmaintained so i guess it makes sense to have our own fork with the fixes

lucifer: you can append `.0` to the 2nd line FROM… for testing

(in `build/postgres/Dockerfile`)

makes sense, trying

beta updated, now updating prod

prod is updated

Thanks!

So I guess we do a release on Monday

Should we look into AC stuff or EAA next?

yes ;)

Putting both on beta at once seems a bit much

both will probably need an extra week or two in beta

I guess it depends on how long the beta period is

The EAA thing in test does seem better now, anyway.

All the issues we found seem correct

kellnerd: thank you for the blog check! I knew you would find things to improve 🫶

!m yvanzo for keeping users updated on all channels 🙌

You're doing good work, yvanzo for keeping users updated on all channels 🙌!

!m team for sorting out docker!!

You're doing good work, team for sorting out docker!!!

(at least, from the chat I gather that it’s been sorted?)

looks to be sorted, thankfully

amazing, what a frikkin headache

[musicbrainz-docker] 14yvanzo opened pull request #268 (03master…fix-pg-amqp-build): Fix building pg_amqp with debian bookworm https://github.com/metabrainz/musicbrainz-docke...

bitmap: Has production been updated to the next release? If so, we can announce it tomorrow.

I only merged the cache PR into what was already in production

yvanzo: got the same error documented in the summit notes doc

bitmap: Ok thus the release on Monday, I marked the ticket in beta and updated the beta banner accordingly.

thanks!

lucifer: as expected, let’s look closer into this tomorrow then :)

just checked and all container IPs and MACs match, aside from app on mingus but that seems intentional

yvanzo: sure, makes sense

I noticed an issue in the cache, but I don't think it's a cause of any bugs...just some bogus keys being set

MB:release:recently_invalidated:HASH(0x55e14656ac08)

atj, zas: I didn’t remove the containers on mingus because I wasn’t sure how to re-create these.

it seems to occur when a release is merged

I guess I could look into that tomorrow, I'd rather review derat's PR tonight

yvanzo: no harm AFAICS

yvanzo: does that seem odd to you? https://stats.metabrainz.org/d/000000058/redis-...

I can manually trigger a BGREWRITEAOF, but I'm not sure why it's not doing that automatically anymore (?). and the pattern seems to have had already changed before I deployed the cache PR

well, I went ahead and ran that (at 23:04 UTC), as you can see in the graph