#metabrainz

[listenbrainz-server] 14samglt opened pull request #3072 (03master…patch-1): Docs: Update link to Java client https://github.com/metabrainz/listenbrainz-serv...

<aerozol[m]> "Oh, about removing akshaaatt..." <- Thanks! Got busy with some clients currently. Racing through life. Sorry for missing the meetings. Will be back soon and let you know! 🙌⚡️

All good, no stress!! Glad you're busy with clients 😝

lucifer: there's no disk space left on marlon, since yesterday around 9:20 UTC

Most used space is in /data/spark-tmp

I reckon that would be the cause of stats not being calculated

[metabrainz.org] 14amCap1712 merged pull request #492 (03master…lb-scope): Add ListenBrainz listen submission scopes https://github.com/metabrainz/metabrainz.org/pu...

zas: not sure what was occupying the space but restarting the worker cleared it up automatically. i wasn't able to check what was there, will take a look again tomorrow after running some jobs to see what is piling up.

zas: can you share gateways logs for metabrainz-oauth-prod container?

monkey: could you do me a favor and post to Masto/bday/community about the spark cluster and that we know things are borked?

I haven't logged into the socials yet but I can try

<mayhem[m]> "monkey: could you do me a..." <- OK, doneskies

Danke!

monkey: great and very informative response on the forum for the "My issue with the latest blog post", thanks for that 👏

Cheers! :) Complex topics that are above my own head...

:D yes, but I think this still makes it understandable to everyone

not the recommendation algo, I mean, but the problems discussed

Good, I wasn't sure, thanks for the feedback

[listenbrainz-android] 14hemang-mishra opened pull request #503 (03main…Fix-LightthemeFixForListenScreen-hemang_mishra): Fix: Light theme on profile screen and ui enhancement. https://github.com/metabrainz/listenbrainz-andr...

[troi-recommendation-playground] 14MonkeyDo opened pull request #155 (03main…spotify-truncate-description): LB-1629, LB-1699: Truncate and clean up playlist description and title before exporting to Spotify https://github.com/metabrainz/troi-recommendati...

bitmap: hi! do you know how to view pgbouncer stats?

trying to debug the following error

psycopg2.OperationalError: server closed the connection unexpectedly

This probably means the server terminated abnormally

before or while processing the request.

and https://stats.metabrainz.org/d/s2GFFRJZk/pgboun... doesn't seem to work

jasje: Hello! I waited a while to be sure, but the LB android app is still not detecting every song I play, only about 1/3. I switched back to Pano Scrobbler :/

Not sure if it's only my phone, or a more general problem.

Happy to help debug what is happening, we had a good session during the summit !

lucifer: hey, this is the dashboard I would use but I'm not sure who there's no data for metabrainz_db, do you know zas?

s/who/why/, s/metabrainz_db/metabrainz\_db/

i was talking with zas about it earlier but he thinks it has been broken for a while and we were wondering if you knew of another way

not in grafana, there is show stats in the pgbouncer DB but that only gives a snapshot, it doesn't have any historical stats

and only very high-level stats are logged periodically in the pgbouncer logs

docker logs --since '2024-12-09T00:00:00Z' pgbouncer-master does show a lot of issues related to metabrainz_db though, a lot of reserve connections being used and server conn crashed

sounds like a lot of processes were being spawned and eating up too many connections, this could just be related to the default pool size

<monkey[m]> "jasje: Hello! I waited a while..." <- > <@monkey:chatbrainz.org> jasje: Hello! I waited a while to be sure, but the LB android app is still not detecting every song I play, only about 1/3. I switched back to Pano Scrobbler :/... (full message at <https://matrix.chatbrainz.org/_matrix/media/v3/...>)

lucifer: https://github.com/metabrainz/docker-postgres-c... which pgbouncer service are you connecting to?

jasje[m]: If memory serves, when we were looking together at Pano Scrobbler code we realized they had a loop running that kept the app alive, or something like that. We thought it was a waste of resources, but if that's the difference it might be a good option to add

monkey: yes that is true, but since its an open source app, there might be some mistakes sometime in the future that could possibly consume battery in background. It would require stricter PR reviews which is not possible considering the barrier of entry is supposed to be beginner friendly.

lucifer: I deployed that and it seems to have helped on the pgbouncer side, let me know if the errors are fixed on the metabrainz side too

bitmap: no idea, but running `/usr/bin/telegraf -config /etc/telegraf/telegraf.conf -config-directory /etc/telegraf/telegraf.d --test --input-filter pgbouncer` on hendrix doesn't list any `metabrainz_db`

so no metrics for it. Can it be something on pgbouncer config side?

it returns metrics for musicbrainz_db pgbouncer postgres template1

it could be, I'm not sure how this is detecting the pools though

ok, I think I found something

ah I see, it's getting stats from the pgbouncer-any instance, I get it should use pgbouncer-master

or we should have separate stats for each of master, slave, any

I added show_commands = ["stats", "pools", "lists", "databases"] and it adds few metrics

> pgbouncer_databases,db=musicbrainz_db,dc=hetzner,host=hendrix,pg_dbname=musicbrainz_db,server=host\=127.0.0.1\ user\=postgres\ port\=65438\ database\=pgbouncer current_connections=7i,disabled=0i,max_connections=0i,min_pool_size=2i,paused=0i,pool_size=60i,reserve_pool=20i 1733847425000000000

> pgbouncer_databases,db=metabrainz_db,dc=hetzner,host=hendrix,pg_dbname=metabrainz_db,server=host\=127.0.0.1\ user\=postgres\ port\=65438\ database\=pgbouncer current_connections=0i,disabled=0i,max_connections=0i,min_pool_size=2i,paused=0i,pool_size=30i,reserve_pool=15i 1733847425000000000

there won't be any stats for metabrainz_db on port 65438 because it doesn't use this pgbouncer at all AFAICT

lucifer, mayhem : created for you playlists haven't been generated yesterday AFAICT.

not surprising, given the cluster fuckage,

in this case port 65436 should work, but ideally we'd be able to view stats for each service (65436=master, 65437=slave, 65438=any)

monkey: I have some time later, want me to look at LB-1699?

LB-1699: Exporting playlists to Spotify failed with "unknown error" https://tickets.metabrainz.org/browse/LB-1699

Ahaaa! I have a PR open for it in troi, if you could review that I think we should be sorted

ah, ok. will do.

-> truncate title and descriptions as required by spotify + remove newlines

bitmap: I'll add multiple pgbouncer configs, but only port will differ in tags

looks good to me, thanks for getting on that!

atj: around? I may need your help for proper multiple pgbouncer ansible config (I don't think that's supported yet)

[troi-recommendation-playground] 14MonkeyDo merged pull request #155 (03main…spotify-truncate-description): LB-1629, LB-1699: Truncate and clean up playlist description and title before exporting to Spotify https://github.com/metabrainz/troi-recommendati...

julian45: maybe you can help on this too^^

We need to have multiple [[inputs.pgbouncer]] sections as per https://github.com/influxdata/telegraf/tree/mas...

Each with a different address (for ports 65436, 65437, 65438)

I manually did the change in /etc/telegraf/telegraf.d/ansible-inputs.conf on hendrix, but ansible setup has to handle that

mayhem: If you have another minute to look at the improvements in https://github.com/metabrainz/troi-recommendati... , I can aim to release a version of troi

Also mayhem : re. the new service-status endpoint, I think daily jams should be excluded from determining if playlists have run, as most users won't have them (still requires subscribing to troi-bot) and it seems to be running fine even though weekly playlists haven't run. That's the reason we don't have an alert on right now.

I think the right thing to do is to add monitoring for all three types of playlists

(in total)

<zas[m]> "julian45: maybe you can help..." <- I can take a stab at it this evening (my time)

[listenbrainz-server] 14MonkeyDo opened pull request #3073 (03master…youtube-player-state): Fix absent state in YoutubePlayer https://github.com/metabrainz/listenbrainz-serv...

[listenbrainz-android] 1407jasjeet opened pull request #504 (03main…fix-theme): Fix theme https://github.com/metabrainz/listenbrainz-andr...

[listenbrainz-android] 1407jasjeet merged pull request #504 (03main…fix-theme): Fix theme https://github.com/metabrainz/listenbrainz-andr...

[listenbrainz-android] 1407jasjeet merged pull request #503 (03main…Fix-LightthemeFixForListenScreen-hemang_mishra): Fix: Light theme on profile screen and ui enhancement. https://github.com/metabrainz/listenbrainz-andr...

Hey MB team, I’ve just posted about the latest release to the socials. Guessing yvanzo is sick, I feel like he usually does it? I’ve added a notification for MB releases so we should always be across it :)