#metabrainz

no replication lag on pink though

I'm going to try restarting the standby server

there's nothing in the PG logs

I think I found the issue

what is it?

sorry maybe not. the restore_command was incorrect in pink's configuration, but that shouldn't cause WAL files to accumulate...

there were a lot of inserts on floyd

during one hours at 15k ops

found an inactive replication slot on pink, and dropped that, but didn't help either

so many inserts are very likely to cause pink to fall behind floyd

wal increase started right after the end of this event

it makes sense but I'm not seeing any actual replication lag on pink...

it's fully up to date so shouldn't have any need to hold on to these

is it possible it failed to delete those WALs?

it's possible but I didn't find any errors in pink logs

zas: did you clear them just now?

nope; not even logged in

huh. they all just disappeared lol

and pink is in sync with floyd?

maybe it was that inactive replication slot that I dropped and there was a delay in cleaning them up

yeah, same as it was

still nothing in pink logs to indicate wtf it was doing

I'm thinking it was the replication slot I dropped, but I expected it to clean up the WAL files immediately

it's late here, so I'm off to bed, but we need to understand what happened exactly starting with the huge number of inserts that triggered the issue

agreed

if values are correct we got 15k ops for one hour, that's 54M inserts

on musicbrainz_db

the accumulation of wals can be a serious problem if they fill disks

can you link where you see 54M inserts for musicbrainz_db?

from the graph above

check floyd musicbrainzdb inserted

found it, I had the wrong time range

well, that's more 11-12k/s

but still, during one hour that's a lot

so I'm curious about what was inserted, did the database size actually increased?

there was a huge peak in deleted a bit after

it doesn't seem like it

I'm trying to figure out where those operations came from

well, I cannot think anymore, I'm off :) thanks for your help on this, we still have a lot of questions, but the fact wal count stopped increasing is reassuring

gn

good night :)

zas: I think it's traffic from the mbid_mapper (under mapping schema in musicbrainz_db) judging by logs, but not certain

yvanzo: How about something like this for the weblate icon https://usercontent.irccloud-cdn.com/file/QkvYS...

favicon/transparent version https://usercontent.irccloud-cdn.com/file/AdwMB...

I tried using parts of the universal language icon but it’s so complex - because we’re mainly trying to change the favicon I couldn’t get it work. But I used the same hangul letter so there’s some connection

p.s. if we’re keen on that one I’ll double check if it’s okay to turn the character like that

isn't that japanese hiragana?

well I liked it

it seemed right

idk about th colour thoug. is green for translationbrainz? :D

oh god can we name our weblate instance TransBrainz /jk

bitmap: according to time of the load on gaga, it can be https://github.com/metabrainz/listenbrainz-serv...

https://github.com/metabrainz/listenbrainz-serv... ends with a delete operation which could match what we observed

lucifer, mayhem : ^^ can this stuff do ~12k op/s for one hour on musicbrainz_db ?

zas: yes we have a cronjob that runs on thursday and is write intensive.

i am not sure how much ops it generates though

lucifer: too much ;)

zas: i think we can make that particular table unlogged so that its contents aren't written to WAL.

i'll look into its implications and discuss it with mayhem later today.

[listenbrainz-android] 14akshaaatt merged pull request #133 (03main…trackbar): Feat: SeekBar in listenPlayer https://github.com/metabrainz/listenbrainz-andr...

ok, in short, for us, high number of ops on pg floyd, caused creation of 120gb of WAL files on pink (because it couldn't cope with the rate apparently), then new WAL files kept accumulating

we were far from disk size limits, and we're still unsure whether WALs got cleaned after bitmap's action or not

also it got almost unnoticed because of a buggy alert (fixed now, the good status of floyd was hiding the bad status of pink)

i see makes sense. i think we should be able to avoid the WAL creation atleast with the UNLOGGED quickfix.

lucifer: it's possible to run this task at anytime right? I mean can we trigger it for real-life testing (in order to verify it has expected results, but under control this time)?

zas: yes, we can.

we measured a lot of db inserts (for more than one hour), followed by a lot of delete operations under a short time. Can you link those to code?

wal counts on pink & floyd (from https://stats.metabrainz.org/goto/WNNWZRlVR?org...)

CatQuest: looks Japanese for sure! But pretty sure it's Korean script.. they are all so similar!

The green is just the MetaBrainz color, I've been using it for stuff that covers all projects (tickets, forums, wiki)

I like the icon :)

moooin

lucifer: zas : not logging the metadata cache makes sense to me. its all derived data.

mayhem: morning

do you have any idea why the problem only appears now? I wonder if that's a change on lb side or due to the switch floyd<->pink we did earlier this week

I mean this cron job runs since a long time, right?

yes, no change on our side.

awww man, 0 listens recorded for all day yesterday. I'm guessing the spotify app dropped off at some point and then panoscrobbler couldn't do its job.

so panoscrobbler isn't up to the task either.

hmmm, it happened in the past, last month we had a similar peak on pink, though it didn't had the same results

both peaks are very similar, (actually 2, inserted then deleted)

but the issue appeared after the switch to floyd

bitmap: ^^

we had an alert on WAL count last month, just after the peak, but it recovered

Both events compared https://usercontent.irccloud-cdn.com/file/Nv3j4...

atj: around?

zas: yes for a short while

mayhem: to confirm, unlogging the table implies that it won't be available on replicas and will not survive crash or unclean shutdowns.

its 100% derived data, yes? then I see no problem with that.

yup completely derived data. cool, sounds good.

atj: will you be available for the move to new consul today?

i have work commitments until 13:30, then lunch, so will be around after 14:00

can you list the steps that are required for the migration?

lucifer, mayhem: even though it triggered the pink WALs issue, I don't think that's an actual issue on LB side. Of course, reducing the use of main db is always better, but this process didn't trigger the issue on last month run.

atj: yes, that's quite short: deploy consul/unbound changes, stop dnsmasq & consul-agent containers, restart consul/unbound with new configs

well, we need to test the process manually on few nodes first, but I think it can be automated

once done, we can totally remove few containers from each node (serviceregistrator-10.2.2.*, consul-agent, dnsmasq) and remove them from startup scripts

now, we hope the deployment doesn't break anything in running containers, I expect surprises as usual

do we want to do this on a Friday afternoon?

bit of a tortured metaphor there ;) are we expecting a flood?

well, yes, always ;)

more seriously we can delay that to Monday morning

i'd like to do the testing and write the playbook this afternoon

ok, and we can refine steps

then we can roll it out on Monday

ok for me

zas: have you started blocking lidarr?

rudi/rex are blocking over 500 HTTPS req/s

damn.

lidarr specific or total?

total

i was just surprised by the numbers

yeah, QNAP et al.

atj: dreq is only what we block based on IPs, blocking based on UAs is done at openresty level (but I parse logs from time to time to block some at IP level)

right, you can't block UAs at TCP level

mayhem: I blocked lidarr-extended for now

I think we should block anything with lidarr in its UA. they have their own hosting -- there is no need to serve any of their requests.

ok, I'll check logs

I see no request from lidarr (only from lidarr-extended)

ok. lets keep monitoring.

oh wait, there are few, very few

I'll block those too

great

done

thx

I did not know zas is the new Cantona, but maybe the weirdly poetic metaphors are a French people thing :D

:D

Yes they are

mayhem, aerozol: Recs page has been nicely polished up, and is ready for inspection!

One question I wanna ask you is whether you should be able to save my playlists, or if you should see those save buttons only on your own profile: https://test.listenbrainz.org/user/mr_monnkey/r...

monnkey!

herr

moonkey 🌙🐒

I think a user should be able to save someone else's playlist, yes.