#metabrainz

I fixed a bunch of alerts: scan-exporter was reporting unexpected open/closed ports on floyd, now refreshed

caa tx packets error ->due to a change in telegraf config, fixed, false alerts, CAA ok

critiquebrainz was erroring, I restarted critiquebrainz-web-prod on trille

it was trying to connect to floyd db apparently

^^ this is something cb devs need to have a look imho

bitmap: sir-prod isn't running it seems, I thought yvanzo will move it, but that's not the case, or something is broken in alerts

I restarted sir-prod on floyd, container wasn't running

ok, it seems to work again, all alerts cleared

zas: sorry was eating, thanks for taking care of it

btw sentry has been acting up for a while, the logs have some error related to redis

wasn't sure how to deal with that. (tried restarting redis, but it looks like it might be stuck starting up, since the web container just returns "Redis is loading the dataset in memory" for everything)

on which host it is?

btw, are you aware of Patroni? I discovered it reading https://jfrog.com/community/devops/highly-avail...

65.108.55.50 (dunno what the hostname is, syswiki says sentry-cloud.metabrainz.org but that doesn't resolve for me)

yeah, I've heard of Patroni

no idea about sentry stuff, I guess alastairp can have a look in few hours

it looked interesting but repmgr involved a lot less magic to get working iirc :) (and we were already using barman from 2ndquadrant)

I'm off to bed (3AM soon)

there is actually a haproxy container in the docker-postgres-cluster repo which tries to automatically route requests to the current primary, it's not tested but I'd like to try it out soon

have a good night!

thx

zas: alastairp: are ‘CB devs’ just alastair? Maybe? Just making sure you see the error zas posted above alastair, sorry ;)

monkey: for when you’re back, just thought this was interesting, and looks very LB already. Do we want to do something similar eventually? https://album.link/b/1646963467

It also makes me think that we’ve had mockups where we try squish in genres/tags in LB, but we could just do them quite big instead. Not worry about really taking up space, being mobile friendly (but only letting one listen expand at a time?)

!m zas the nightworker

You're doing good work, zas the nightworker!

aerozol: lucifer has CB’s back ^

Phew

bitmap, yvanzo: can we try to get https://github.com/metabrainz/musicbrainz-serve... merged relatively soonish? (obviously once we're done with the relationship editor)

We've gotten a bunch of crypto scam edit notes that would be good to purge

There is a lot to say about this PR but I’m available to help with handling the current scam through database updates if needed.

For now, it's small enough we probably don't need to rush into it - but if it gets bigger before we have a proper way of doing this we can look into removing them via the DB :)

moooin!

aerozol: yes, definitely have been dreaming about a good album page on LB, if that's what you mean.

monkey: even in the individual ‘listen’ drop down on the home page, but album page too!

aerozol: what part of this link page would you see in the listen card dropdown?

Buy and listen but Copy URL/Embed/Share is nice too

But I was mainly thinking about the design rather than the actual content

Nice and spacious

good night by the way :D

hello, looking at chat history / CB stuff now, thanks for the ping

lucifer: hi, I was talking to mayhem earlier in the week and we were thinking that it'd be a good idea to have a primary key column in the listen table (for when you're going to rewrite it). The reason is that we're probably going to start publishing other datasets of things like messybrainz id mappings, and it'd be useful to do things like indicate which listens a particular messybrainz id maps to

while it's possible to have a unique id on user/name/date (or user/msid/date in the future), that'd make such a dump much larger if we had to include all fields to identify a unique key

zas: regarding CB database connection, I'm a bit surprised at that error. We use the pgbouncer-master service in consul: https://github.com/metabrainz/critiquebrainz/bl... and consul-template is configured to restart the service after a re-render of the config file: https://github.com/metabrainz/critiquebrainz/bl...

are you able to confirm about what server 10.2.2.44 was at that time? Maybe there was a hiccup in consul, or maybe it hadn't restarted yet?

regarding sentry-cloud, this was the DNS name that we gave the VM temporarily while sentry.meb.org was still pointing to gateways. I've updated syswiki to point to the proper hostname

alastairp: floyd is 10.2.2.44

10.2.2.44 didn't change, it was floyd, but pg master moved to pink, it was solved after I restarted the container, so I guess it didn't actually restarted on template change or smt like that

strange

[listenbrainz-android] 1407jasjeet opened pull request #47 (03main…Mobile-117-jasje): Added lifecycle check https://github.com/metabrainz/listenbrainz-andr...

I'm not 100% certain of the sentry redis error that bitmap said, I see that sentry is up at the moment, but I do see this error that he mentioned in the webserver logs.

redis DB is currently 800mb, that doesn't seem very big to me (VM has 16gb memory)

pgbouncer-slave service on floyd https://www.irccloud.com/pastebin/ktrDzzyz/

on pink https://www.irccloud.com/pastebin/QJnilSRL/

I wonder if we had 2 pgbouncer-master at the same time, may explain the issue

https://develop.sentry.dev/self-hosted/troubles... "Redis is used both as a transactional data store and a work queue of Celery in the self-hosted setup. For this reason, it may get overwhelmed during event spikes."

it's possible that the db switchover caused many errors to be sent to sentry and redis got overwhelmed

very likely

alastairp: too bad we don't have logs anymore for cb web container, they were spammed and since they are limited in size we don't have lines for when the switch occured

akshaaatt: please update sentry to v3.4.2 where the app crashing is fixed.

we can upgrade to room 2.5.0 after that

lucifer: from cron dumps log, "2023-02-03 00:00:02,800 listenbrainz.webserver ERROR Invalid dump ID 1191, could not find previous dump"

is this possibly related to the postgres master switchover, causing a jump in the PK sequence?

I hear MB is mentioned here: I haven't listened to it yet... https://open.spotify.com/episode/63lTbBpT0IJtJh...

I also heard that it was mentioned here: https://open.spotify.com/track/4cOdK2wGLETKBW3P...

alastairp: PK makes sense. about dumps, yeah its likely due to switchover.

not sure if this is the right place but i have a physical cd that i was going to rip to flac. the album is in musicbrainz but the discid wasnt associated with it so i submitted that. the problem is now that the artists real name is used rather than his performance/artist name. whats the best/correct way to change the artist name for an album?

specifically, the album is "The Second Coming" by "Roger Troutman II" (his artist/performer name), but in musicbrainz its listed at "The Second Coming" by "Roger Troutman Jr." (his real name)

}8]: you should be able to get some feedback on this question in the #musicbrainz channel

ok thanks

(while there is an overlap in the membership of both channels, this one tends to focus more on the software development of musicbrainz, while that one has more editors of the database)

ahh ok, sorry, i didnt realize that

no problem at all!

Does BrainzBot has backups? If not, the data volumes of the new dockerized instance could be collected by borg backup.

alastairp: By deploying on the current Postgres instance, it seems that you just addressed OTHER-319, right?

OTHER-319: Update BrainzBot dependencies https://tickets.metabrainz.org/browse/OTHER-319

yvanzo: yes, this is in the middle of happening. By moving to MeB infra we have chatlogs (and botbot configuration) in the main PG cluster, which gives us automatic backups using the current backup system.

There is a redis server, but I don't think it's useful to back up its volume using borg, it's just used as a transient message queue between the chatbot and plugins

👍

in terms of python dependency upgrades, I've already applied Leo's changes to upgrade to django 1.11 and python 3, with a pending task to upgrade to django 2.2. I've also started chopping a lot of code out of the original chatlots webapp from the original upstream project that we forked from (no longer maintained)

I think we've managed to remove approximately half of the dependencies, which makes ongoing maintenance much easier

unfortunately there is one large dependency used to generate bootstrap html from python which isn't supported with the latest version of django, so we can't yet upgrade to that

However we've been talking for years about how the web interface is terrible, and this may be a chance to rewrite the interface that generates the chat logs allowing us to make something simpler that doesn't have bad javascript, and at the same time reduce the number of external dependencies that the app has

bitmap, yvanzo: hi! :)

alastairp: oh, something that doesn't suck would be nice

reosarevok: taking feedback for ideas

"something that doesn't suck" is a good starting point ;D

but honestly, 1 URL per day with a prev/next button, and real working #anchors is probably 100x better than what we currently have

or even just static html per month like what we used to have :-P (that being said, I do like the search)

+1 - only thing we'd need is the search

yeah, search is the one thing that makes a fully static system more difficult

personally I think the search works quite well - itd just be great if the results actually linked to the message :-P

alastairp: Thanks, I’ve assigned the ticket to you to reflect the current status. Feel free to create sub-task if needed.

thanks yvanzo

reosarevok, yvanzo hello

hi you both

The banner text is gone from the beta website, do you have it somewhere?

A tip is to comment banner text with <!-- --> for temporarily displaying a different message.

yes, just added it back

Thank you

yesterday I spent well over and hour investigating those "Two relationships with the same key" errors and didn't get anywhere, so I'm working on improving the error message to give more information

So there are 68 tickets addressed in beta, and 70 tickets reporting about beta.

Seems like a good plan

Which is really good to have that much feedback for beta :)

There's also a bunch of useful comments on https://community.metabrainz.org/t/help-test-th...

(new ones)

https://community.metabrainz.org/t/help-test-th... for example is a funny issue, although not surprising

heh. yes, I saw some of those

It’s getting wild, it would probably help to move it to tickets (existing or new ones)?

The discussion being there is good (not everyone can be bothered to comment on tickets) but we should probably create tickets for some of these

tickets would help make tracking the issues a bit easier for me. I'm not sure how many of these will be fixed before monday, though

It’s good to have feedback for sure, it’s better to have it in a structured way issues can be tracked, for example with... an issue tracker?

i'd prioritize the ones that crash the page or throw errors

If we need a bit of time to try to debug the clashing rels error, maybe we should keep this in beta one more week anyway?

I can enter some tickets based on the feedback, at least

it's better to have an extra week than rush it, I'd say

bitmap: Do you want us to start with helping on a specific issue you are stuck with? reviewing the existing tickets? or referencing the issues mentioned in this thread as tickets?

(Or any other way we can help with getting a more clear idea of what is left to be done or to be postponed)

I was mainly stuck on MBS-12860 and MBS-12861 yesterday but it seems like they are not easy to reproduce without more info. help with opening issues from comments in the thread would be helpful

MBS-12860: beta: Unable to enter relationships: "Two relationships with the same key" https://tickets.metabrainz.org/browse/MBS-12860

MBS-12861: Beta: "Error: Two relationships with the same key" does not specify the problematic relations https://tickets.metabrainz.org/browse/MBS-12861

If you can fix 12861 (so, make the error give better info) then by having it in beta a bit longer we might get more info from users who hit it :)

yup. unfortunately these errors weren't logged to sentry and I'm not sure why

We should merge https://github.com/metabrainz/musicbrainz-serve... and https://github.com/metabrainz/musicbrainz-serve... before we release too

(happy to make changes on the second if needed)

we do have a captureExceptions call for any errors that occur

so either sentry was down, or blocked by both users, or the error capturing didn't work

IIRC we've had the impression at least one of the users does block sentry before?

Ok, I’m starting with “opening issues from comments in the thread”. I will mention each post here before opening a ticket, and edit the post to link to the ticket after.

yvanzo: ok, you start from the top I start from the bottom? : )

👍

one of them might but I've seen errors in sentry for the other before

Hmm, then maybe there's a second issue with the logging, ok

"The never ending “show more”. Would be much better to show how many more there actually are in the list. “Show more (10 items)” or similar." bitmap: do you know where we show "show more"? https://beta.musicbrainz.org/artist/dc8c2046-46... already shows "See all 105 relationships", but maybe that's in the release rel editor?