17:30 PM
yvanzo
MeB accounts
2024-02-08 03900, 2024
17:31 PM
Satyaraj[m]
> <@irc_yvanzo:moviebrainz.org> tweeted about it
2024-02-08 03900, 2024
17:31 PM
Satyaraj[m]
* your twitter username?
2024-02-08 03912, 2024
17:31 PM
yvanzo
2024-02-08 03926, 2024
17:31 PM
monkey posted on BB twitter
2024-02-08 03928, 2024
17:31 PM
yvanzo
(links)
2024-02-08 03922, 2024
17:32 PM
zas
all alerts cleared (but load on kiss, soon)
2024-02-08 03937, 2024
17:32 PM
yvanzo
lucifer: might be worth updating docker-server-configs then
2024-02-08 03909, 2024
17:33 PM
lucifer
yvanzo: it was a temporary move, reverted.
2024-02-08 03931, 2024
17:33 PM
lucifer
containers are now in sync with what is mentioned in docker-server-configs
2024-02-08 03913, 2024
17:34 PM
yvanzo
👍
2024-02-08 03936, 2024
17:35 PM
zas
everything looks good on my side, alerts are all gone, and stats look normal now
2024-02-08 03954, 2024
17:35 PM
bitmap
nice, same here
2024-02-08 03901, 2024
17:36 PM
monkey
👏
2024-02-08 03949, 2024
17:36 PM
bitmap
I'll bbiab but can look into deploying MBS-13449 afterward, yvanzo
2024-02-08 03950, 2024
17:36 PM
BrainzBot
2024-02-08 03945, 2024
17:37 PM
zas
We finally sorted out this mess. Now back to real work ;)
2024-02-08 03908, 2024
17:38 PM
yvanzo[m] joined the channel
2024-02-08 03908, 2024
17:38 PM
yvanzo[m]
zas: Where is the script to double-check that there is no longer any duplicate MAC addresses?
2024-02-08 03956, 2024
17:38 PM
zas
ansible -m shell -a 'docker ps --format {% raw %}{{.ID}}{% endraw %} | while read id; do docker inspect $id | jq ".[] .NetworkSettings .Networks .bridge .MacAddress"; done |sort |uniq -c |sort -n |grep -vE "^\s+1|null" || true' docker_servers
2024-02-08 03906, 2024
17:39 PM
zas
I just checked, all clear
2024-02-08 03918, 2024
17:39 PM
yvanzo
👏
2024-02-08 03947, 2024
17:41 PM
zas
lucifer: on kiss & gaga it would be great to prune old containers & images
2024-02-08 03919, 2024
17:42 PM
zas
bitmap: I just did on paco, because of jenkins (it creates images but never clear them, reclaimed 5gb)
2024-02-08 03951, 2024
17:42 PM
zas
lucifer: I mean kiss & wolf (gaga's ok)
2024-02-08 03940, 2024
17:44 PM
mayhem[m] joined the channel
2024-02-08 03940, 2024
17:44 PM
mayhem[m]
bitmap (IRC): just got mail from matthew@kite about a missing repllication packet on
meb.org . could you please look into that?
2024-02-08 03940, 2024
17:44 PM
mayhem[m]
2024-02-08 03941, 2024
17:50 PM
bitmap
mayhem[m]: it's expected since cron was stopped after packet 165383 from 15:00 UTC. 165384 will be generated in ~10 minutes
2024-02-08 03959, 2024
17:50 PM
mayhem[m]
fabu, thanks!
2024-02-08 03907, 2024
17:53 PM
lucifer
zas: done
2024-02-08 03926, 2024
17:53 PM
zas
lucifer: thanks!
2024-02-08 03937, 2024
17:53 PM
mayhem[m] relays the message to the BBC/kite
2024-02-08 03904, 2024
17:55 PM
bitmap
I should have timed that better and let cron run at 16:00, will consider that next time
2024-02-08 03904, 2024
17:57 PM
yvanzo
lucifer: What time would work for you on tomorrow?
2024-02-08 03928, 2024
17:57 PM
lucifer
yvanzo: i should be around most of the day.
2024-02-08 03907, 2024
17:58 PM
yvanzo
Ok, I should be up early but leave early too.
2024-02-08 03921, 2024
17:58 PM
lucifer
cool, ping me whenever you are around.
2024-02-08 03933, 2024
17:58 PM
yvanzo
👍
2024-02-08 03946, 2024
17:58 PM
yvanzo
bitmap: Do you mean downtime at :15?
2024-02-08 03954, 2024
18:35 PM
ApeKattQuest
åssen gor'e?
2024-02-08 03958, 2024
18:35 PM
ApeKattQuest
åssen går'e?
2024-02-08 03933, 2024
18:37 PM
lucifer
yvanzo: can you point me to the branch or instructions to follow? I have some time now too and can test it out.
2024-02-08 03946, 2024
18:37 PM
bitmap
yvanzo: sorry not sure I understand the question
2024-02-08 03954, 2024
18:38 PM
yvanzo
2024-02-08 03941, 2024
18:39 PM
yvanzo
lucifer: Instructions are the same but using Git Bash
2024-02-08 03919, 2024
18:40 PM
lucifer
yvanzo: can I use wsl 2 Ubuntu bash instead?
2024-02-08 03937, 2024
18:40 PM
yvanzo
sure if you want
2024-02-08 03954, 2024
18:40 PM
lucifer
cool
2024-02-08 03925, 2024
18:41 PM
yvanzo
2024-02-08 03954, 2024
18:42 PM
yvanzo
bitmap: scheduling downtime at :15 would let replication packets go next time :)
2024-02-08 03937, 2024
18:43 PM
bitmap
ah, right
2024-02-08 03916, 2024
18:59 PM
BrainzGit
2024-02-08 03959, 2024
19:03 PM
bitmap
I'm merging production into the beta branch now, is there anything else I should include (or is beta frozen?)
2024-02-08 03946, 2024
19:09 PM
bitmap just merges production only
2024-02-08 03913, 2024
19:10 PM
relaxoMob has quit
2024-02-08 03914, 2024
19:15 PM
reosarevok
I'm not sure tbh
2024-02-08 03930, 2024
19:15 PM
reosarevok
yvanzo would know
2024-02-08 03912, 2024
19:17 PM
bitmap
I'm just deploying the cache changes for now
2024-02-08 03933, 2024
19:17 PM
reosarevok
Sure
2024-02-08 03930, 2024
19:26 PM
yvanzo
👍
2024-02-08 03918, 2024
19:27 PM
lucifer
yvanzo: pg_amqp is failing to build
2024-02-08 03926, 2024
19:27 PM
lucifer
2024-02-08 03930, 2024
19:27 PM
yvanzo
beta is frozen for a while, only #3130 was missing for a simultaneous beta/prod update
2024-02-08 03932, 2024
19:29 PM
bitmap
ok, updating beta now
2024-02-08 03934, 2024
19:29 PM
yvanzo
lucifer: is it for `docker-compose build --no-cache db`
2024-02-08 03905, 2024
19:30 PM
lucifer
yvanzo: `docker-compose build`
2024-02-08 03911, 2024
19:30 PM
lucifer
i can try building just db too.
2024-02-08 03921, 2024
19:30 PM
lucifer
but yes this failure was for db image.
2024-02-08 03906, 2024
19:31 PM
yvanzo
lucifer: it's failing here too
2024-02-08 03946, 2024
19:31 PM
huhridge has quit
2024-02-08 03939, 2024
19:32 PM
yvanzo
probably due to some changes upstream in postgres:12 image
2024-02-08 03953, 2024
19:32 PM
lucifer
ah :/
2024-02-08 03904, 2024
19:33 PM
lucifer
they seemed to have moved to bookworm.
2024-02-08 03916, 2024
19:33 PM
lucifer
not sure if that's new
2024-02-08 03922, 2024
19:37 PM
lucifer
do you know if we use bookworm or bullseye in production?
2024-02-08 03914, 2024
19:38 PM
bitmap
for the main cluster?
2024-02-08 03923, 2024
19:38 PM
petitminion joined the channel
2024-02-08 03947, 2024
19:38 PM
lucifer
yes
2024-02-08 03913, 2024
19:39 PM
bitmap
they use metabrainz/base-image which is based on ubuntu jammy
2024-02-08 03938, 2024
19:39 PM
lucifer
ah interesting, and install postgres using apt?
2024-02-08 03959, 2024
19:39 PM
bitmap
yeah from the PGDG repo
2024-02-08 03909, 2024
19:40 PM
bitmap
2024-02-08 03949, 2024
19:41 PM
bitmap
I vaguely remember having to patch pg_amqp to add "int" to all of the broker_id arguments, but I don't remember what for.
2024-02-08 03911, 2024
19:42 PM
lucifer
yup that is what these warnings show as well
2024-02-08 03923, 2024
19:42 PM
lucifer
do we have a fork of pg_amqp?
2024-02-08 03934, 2024
19:42 PM
bitmap
nope
2024-02-08 03913, 2024
19:43 PM
bitmap
this might've been for the postgres-json-dump container, which uses the postgres:16.1-bookworm image
2024-02-08 03935, 2024
19:43 PM
lucifer
the official repo looks more or less unmaintained so i guess it makes sense to have our own fork with the fixes
2024-02-08 03900, 2024
19:45 PM
yvanzo
lucifer: you can append `.0` to the 2nd line FROM… for testing
2024-02-08 03924, 2024
19:45 PM
yvanzo
(in `build/postgres/Dockerfile`)
2024-02-08 03917, 2024
19:46 PM
lucifer
makes sense, trying
2024-02-08 03925, 2024
19:46 PM
bitmap
beta updated, now updating prod
2024-02-08 03907, 2024
20:06 PM
Zhele has quit
2024-02-08 03951, 2024
20:17 PM
Zhele joined the channel
2024-02-08 03930, 2024
20:18 PM
bitmap
prod is updated
2024-02-08 03901, 2024
20:20 PM
reosarevok
Thanks!
2024-02-08 03916, 2024
20:20 PM
reosarevok
So I guess we do a release on Monday
2024-02-08 03952, 2024
20:20 PM
reosarevok
Should we look into AC stuff or EAA next?
2024-02-08 03910, 2024
20:22 PM
bitmap
yes ;)
2024-02-08 03953, 2024
20:22 PM
reosarevok
Putting both on beta at once seems a bit much
2024-02-08 03925, 2024
20:26 PM
petitminion has quit
2024-02-08 03930, 2024
20:26 PM
bitmap
both will probably need an extra week or two in beta
2024-02-08 03922, 2024
20:27 PM
bitmap
I guess it depends on how long the beta period is
2024-02-08 03900, 2024
20:31 PM
Satyaraj[m] has quit
2024-02-08 03901, 2024
20:32 PM
reosarevok
The EAA thing in test does seem better now, anyway.
2024-02-08 03908, 2024
20:32 PM
reosarevok
All the issues we found seem correct
2024-02-08 03901, 2024
20:38 PM
yvanzo[m] has quit
2024-02-08 03906, 2024
20:40 PM
relaxoMob joined the channel
2024-02-08 03900, 2024
20:49 PM
aerozol
kellnerd: thank you for the blog check! I knew you would find things to improve 🫶
2024-02-08 03940, 2024
20:50 PM
aerozol
!m yvanzo for keeping users updated on all channels 🙌
2024-02-08 03940, 2024
20:50 PM
BrainzBot
You're doing good work, yvanzo for keeping users updated on all channels 🙌!
2024-02-08 03957, 2024
20:50 PM
aerozol
!m team for sorting out docker!!
2024-02-08 03957, 2024
20:50 PM
BrainzBot
You're doing good work, team for sorting out docker!!!
2024-02-08 03909, 2024
20:51 PM
aerozol
(at least, from the chat I gather that it’s been sorted?)
2024-02-08 03937, 2024
20:53 PM
mayhem[m] has quit
2024-02-08 03940, 2024
20:55 PM
petitminion joined the channel
2024-02-08 03938, 2024
20:57 PM
bitmap
looks to be sorted, thankfully
2024-02-08 03932, 2024
20:59 PM
aerozol
amazing, what a frikkin headache
2024-02-08 03957, 2024
20:59 PM
BrainzGit
2024-02-08 03938, 2024
21:01 PM
yvanzo
bitmap: Has production been updated to the next release? If so, we can announce it tomorrow.
2024-02-08 03957, 2024
21:01 PM
yvanzo
Will have to fix building PG for mirrors though. ^
2024-02-08 03957, 2024
21:02 PM
bitmap
I only merged the cache PR into what was already in production
2024-02-08 03904, 2024
21:04 PM
lucifer
yvanzo: got the same error documented in the summit notes doc
2024-02-08 03958, 2024
21:13 PM
yvanzo
bitmap: Ok thus the release on Monday, I marked the ticket in beta and updated the beta banner accordingly.
2024-02-08 03916, 2024
21:14 PM
bitmap
thanks!
2024-02-08 03912, 2024
21:15 PM
yvanzo
lucifer: as expected, let’s look closer into this tomorrow then :)
2024-02-08 03922, 2024
21:18 PM
atj
just checked and all container IPs and MACs match, aside from app on mingus but that seems intentional
2024-02-08 03942, 2024
21:18 PM
lucifer
yvanzo: sure, makes sense
2024-02-08 03917, 2024
21:19 PM
bitmap
I noticed an issue in the cache, but I don't think it's a cause of any bugs...just some bogus keys being set
2024-02-08 03927, 2024
21:19 PM
bitmap
MB:release:recently_invalidated:HASH(0x55e14656ac08)
2024-02-08 03942, 2024
21:19 PM
yvanzo
atj, zas: I didn’t remove the containers on mingus because I wasn’t sure how to re-create these.
2024-02-08 03944, 2024
21:19 PM
bitmap
it seems to occur when a release is merged
2024-02-08 03955, 2024
21:20 PM
bitmap
I guess I could look into that tomorrow, I'd rather review derat's PR tonight
2024-02-08 03910, 2024
21:21 PM
atj
yvanzo: no harm AFAICS
2024-02-08 03922, 2024
21:42 PM
bitmap
2024-02-08 03911, 2024
22:29 PM
relaxoMob has quit
2024-02-08 03928, 2024
22:30 PM
relaxoMob joined the channel
2024-02-08 03933, 2024
22:58 PM
bitmap
I can manually trigger a BGREWRITEAOF, but I'm not sure why it's not doing that automatically anymore (?). and the pattern seems to have had already changed before I deployed the cache PR
2024-02-08 03958, 2024
23:05 PM
bitmap
well, I went ahead and ran that (at 23:04 UTC), as you can see in the graph
2024-02-08 03938, 2024
23:29 PM
huhridge joined the channel
2024-02-08 03925, 2024
23:34 PM
huhridge has quit
