#metabrainz

[listenbrainz-server] 14amCap1712 opened pull request #2617 (03master…spark-base-image): Update spark docker image to build pycurl and urllib3 https://github.com/metabrainz/listenbrainz-serv...

thx!

this shouldn't change anything but lets' see

because the actual image was built and pushed to dockerhub 2 days ago

hm. random CB tests failing on my pr. seems unrelated.

lucifer: spark tests still failing. I merged your branch into mine

mayhem: yes i the previous runs, its a new error. looking into it.

*saw the

reosarevok: had to get this out of my head, been floating around for months… if we could assign a ‘product’ attribute to works then I think we could display something like this without having to get new data? It’s already all there, just doesn’t display in a way that’s useful for me https://usercontent.irccloud-cdn.com/file/bT0ub...

The niggly bit would be the year, but this would be useful for me even if that column didn’t exist

[listenbrainz-server] 14mayhem opened pull request #2618 (03master…fix-play-release): LB-1386: Fix play various artists album on LB https://github.com/metabrainz/listenbrainz-serv...

There an issue with LB, I get this when I try to load my dashboard:

ok, shifted work-day, but I still got stuff done. except paying some stragglers.

bitmap: ^^

on prod, monkey ?

yvanzo:

Woopp,s sorry for the ping

I mean Yes

seems related to the DB move.

yep. got the same

zas: atj : bitmap : we've got some DB issues!

I'll stop some of the pgbouncer instances for now

we might have to increase pg's max_connections

MB seems unaffected.

they are not expensive, so why are we so conservative with them?

yeah, we should next time we restart PG

the problem is still happening.

anything I can help with bitmap?

I stopped the pgbouncer on hendrix which should reduce the # of connections

hard to tell what's going on though 'cause I think the pgbouncer stats aren't working since the move

no change. should I restart the web container again?

you can try, I'll try increasing LB's connection allotment in the pgbouncer config

no change. :(

Some alerts on telegram about gaga. Related?

ah, we're looking at the wrong db. :)

bitmap: its gaga, not jimmy/hendrix

lucifer: ping

gaga PG is quite unhappy.

ah, no wonder nothing was taking

pong

gaga's PG is very unhappy.

help me look , please?

sure, unhappy as in?

high load, many PG procs.

this error was found a lot in the logs before we ran out of connections.

that might be eating the connections.

2023-06-06 02:38:15.409 UTC [3991072] STATEMENT:

5 months ago

ok, this was in the logs before.

do we have a spark process or anything writing to gaga right now?

not really

checking pg_stat_activity i see mostly listen read requests

we should check where the connections are coming from.

how do we do that?

its back to normal now i think

did you change anything?

nope

odd. wtf?

16824 | listenbrainz_ts | 1179026 | | | | | | | | 2023-11-02 20:28:19.08004+00 | 2023-11-02 20:42:10.394548+00 | 2023-11-02 20:42:10.394548+00 | 2023-11-02 20:42:10.394548+00 | Timeout | VacuumDelay | active | | 406250699 | autovacuum: VACUUM ANALYZE

_timescaledb_internal._hyper_41_6573_chunk | autovacuum worker

a very wild guess.

plausible.

autovaccum locked listen table, leading to accumulation of connections

but then autovaccum timed out so it went back to normal again

so went from immediate bad to long term bad

Although LB is accessible, it shows that I haven(t listened to anything on my profile

I wonder what spooked autovacuum.

monkey: give it a moment to recover

Oki

it couldn't complete within the timelimit

not sure why

all my listens are accounted for, but they are coming from navidrome

its showing some listens now.

(not for me, aerozol's profile)

tomorrow, i think we should run `VACUUM VERBOSE listen` and it may list if there are any obvious problems.

if not, let's upgrade PG/TS on gaga to latest and then look into tuning it.

good plan

lucifer: my ‘older’ button is not working now, I can’t check if the covers are fixed 🫣

aerozol: yup, its related to the db issues.

“You have listened to 8 songs so far”

uh oh

the older button that is.

Ah okay, you’re onto it. Thanks

covers are broken due to a different db issue

which will take longer to fix

allgood

kind of depressing but I should probably share this https://usercontent.irccloud-cdn.com/file/PAOD9...

yerp

maybe they also don't know about the "save to spotify" option?

Happy to ask follow up questions if wanted, otherwise happy to leave it as “something for users to live with” :)

Since I don’t use Spotify I don’t have any idea what’s meant tbh, I assumed it was the player, but maybe not?

this option exists on a playlist if the user has connected spotify

I wonder if this person knows about it

gaga is still under quite a load.

Right - I don’t even know if they are talking about radio, playing stuff from elsewhere on LB, or submitting listens from Spotify...

doesn't matter.

its brainzplayer flakiness that they are upset about. so do as we do and send playlists to spotify.

mayhem: i tried cancelling autovacuum but it seems it isn't going away that easy.

Okay, I’ll pass that on

lucifer: do you need to cancel it for each chunk? 🙄

yeah

well for the chunks its currently running at the least

20 or sor

Hi, my listenbrainz says I have no listens? It should be well over 100k https://listenbrainz.org/user/calculator.ftvb/

fivesheep: hi! yes, our db is currently under high load so some glitches on the site. the data is still there.

your listens are fine fivesheep. the listen counts got borked.

Ah great thank you for the reassurance!!

the second one looks redundant, i'll drop it

Scared LB users on the forums as well

Thanks for the update & the hard work on resolving this! 🙂

lucifer: the popularity endpoint now returns all nulls except for counts: https://api.listenbrainz.org/1/popularity/top-r...

the data in the DB appears to be fine.

time to restart labs api, methinks.

ah no, that is not labs. huh, odd.

well, all seems normal now, except for playcounts and top-recordings. the latter is not so important now.

mayhem: it's not just the play counts that are borked from a user point of view. no listen is shown that is older than 2h ago, so it looks to users as if they lost all their listens. See mine for example: https://listenbrainz.org/user/mr_monkey/

i understand.

but pagination is borked because of the play counts.

Not just pagination. Even with a manual date of yesterday I can't load yours: https://listenbrainz.org/user/rob/?min_ts=16989...

its all connected, monkey.

So weird

listen counts, minimum timestamps and maximum timestamps are all calculated in the same process.

and without min/max ts you get no working pagination.

What a shock seeing the LB data missing. I really had enough data loss shit this week. But glad to hear the listens themselves are fine. Wish you the best getting everything back up