#metabrainz

/

      • zas
        I fixed a bunch of alerts: scan-exporter was reporting unexpected open/closed ports on floyd, now refreshed
      • caa tx packets error ->due to a change in telegraf config, fixed, false alerts, CAA ok
      • critiquebrainz was erroring, I restarted critiquebrainz-web-prod on trille
      • it was trying to connect to floyd db apparently
      • ^^ this is something cb devs need to have a look imho
      • bitmap: sir-prod isn't running it seems, I thought yvanzo will move it, but that's not the case, or something is broken in alerts
      • I restarted sir-prod on floyd, container wasn't running
      • ok, it seems to work again, all alerts cleared
      • bitmap
        zas: sorry was eating, thanks for taking care of it
      • btw sentry has been acting up for a while, the logs have some error related to redis
      • wasn't sure how to deal with that. (tried restarting redis, but it looks like it might be stuck starting up, since the web container just returns "Redis is loading the dataset in memory" for everything)
      • zas
        on which host it is?
      • btw, are you aware of Patroni? I discovered it reading https://jfrog.com/community/devops/highly-avail...
      • bitmap
        65.108.55.50 (dunno what the hostname is, syswiki says sentry-cloud.metabrainz.org but that doesn't resolve for me)
      • yeah, I've heard of Patroni
      • zas
        no idea about sentry stuff, I guess alastairp can have a look in few hours
      • bitmap
        it looked interesting but repmgr involved a lot less magic to get working iirc :) (and we were already using barman from 2ndquadrant)
      • zas
        I'm off to bed (3AM soon)
      • bitmap
        there is actually a haproxy container in the docker-postgres-cluster repo which tries to automatically route requests to the current primary, it's not tested but I'd like to try it out soon
      • have a good night!
      • zas
        thx
      • aerozol
        zas: alastairp: are ‘CB devs’ just alastair? Maybe? Just making sure you see the error zas posted above alastair, sorry ;)
      • monkey: for when you’re back, just thought this was interesting, and looks very LB already. Do we want to do something similar eventually? https://album.link/b/1646963467
      • It also makes me think that we’ve had mockups where we try squish in genres/tags in LB, but we could just do them quite big instead. Not worry about really taking up space, being mobile friendly (but only letting one listen expand at a time?)
      • serialata has quit
      • yvanzo
        !m zas the nightworker
      • BrainzBot
        You're doing good work, zas the nightworker!
      • yvanzo
        aerozol: lucifer has CB’s back ^
      • aerozol
        Phew
      • jivte_ joined the channel
      • reosarevok
        bitmap, yvanzo: can we try to get https://github.com/metabrainz/musicbrainz-serve... merged relatively soonish? (obviously once we're done with the relationship editor)
      • We've gotten a bunch of crypto scam edit notes that would be good to purge
      • yvanzo
        There is a lot to say about this PR but I’m available to help with handling the current scam through database updates if needed.
      • reosarevok
        For now, it's small enough we probably don't need to rush into it - but if it gets bigger before we have a proper way of doing this we can look into removing them via the DB :)
      • mayhem
        moooin!
      • jivte_ has quit
      • monkey
        aerozol: yes, definitely have been dreaming about a good album page on LB, if that's what you mean.
      • aerozol
        monkey: even in the individual ‘listen’ drop down on the home page, but album page too!
      • monkey
        aerozol: what part of this link page would you see in the listen card dropdown?
      • aerozol
        Buy and listen but Copy URL/Embed/Share is nice too
      • But I was mainly thinking about the design rather than the actual content
      • Nice and spacious
      • good night by the way :D
      • alastairp
        hello, looking at chat history / CB stuff now, thanks for the ping
      • lucifer: hi, I was talking to mayhem earlier in the week and we were thinking that it'd be a good idea to have a primary key column in the listen table (for when you're going to rewrite it). The reason is that we're probably going to start publishing other datasets of things like messybrainz id mappings, and it'd be useful to do things like indicate which listens a particular messybrainz id maps to
      • while it's possible to have a unique id on user/name/date (or user/msid/date in the future), that'd make such a dump much larger if we had to include all fields to identify a unique key
      • zas: regarding CB database connection, I'm a bit surprised at that error. We use the pgbouncer-master service in consul: https://github.com/metabrainz/critiquebrainz/bl... and consul-template is configured to restart the service after a re-render of the config file: https://github.com/metabrainz/critiquebrainz/bl...
      • are you able to confirm about what server 10.2.2.44 was at that time? Maybe there was a hiccup in consul, or maybe it hadn't restarted yet?
      • regarding sentry-cloud, this was the DNS name that we gave the VM temporarily while sentry.meb.org was still pointing to gateways. I've updated syswiki to point to the proper hostname
      • atj
        alastairp: floyd is 10.2.2.44
      • zas
        10.2.2.44 didn't change, it was floyd, but pg master moved to pink, it was solved after I restarted the container, so I guess it didn't actually restarted on template change or smt like that
      • alastairp
        strange
      • BrainzGit
        [listenbrainz-android] 1407jasjeet opened pull request #47 (03main…Mobile-117-jasje): Added lifecycle check https://github.com/metabrainz/listenbrainz-andr...
      • alastairp
        I'm not 100% certain of the sentry redis error that bitmap said, I see that sentry is up at the moment, but I do see this error that he mentioned in the webserver logs.
      • redis DB is currently 800mb, that doesn't seem very big to me (VM has 16gb memory)
      • zas
        pgbouncer-slave service on floyd https://www.irccloud.com/pastebin/ktrDzzyz/
      • I wonder if we had 2 pgbouncer-master at the same time, may explain the issue
      • alastairp
        https://develop.sentry.dev/self-hosted/troubles... "Redis is used both as a transactional data store and a work queue of Celery in the self-hosted setup. For this reason, it may get overwhelmed during event spikes."
      • it's possible that the db switchover caused many errors to be sent to sentry and redis got overwhelmed
      • zas
        very likely
      • alastairp: too bad we don't have logs anymore for cb web container, they were spammed and since they are limited in size we don't have lines for when the switch occured
      • jasje joined the channel
      • jasje
        akshaaatt: please update sentry to v3.4.2 where the app crashing is fixed.
      • we can upgrade to room 2.5.0 after that
      • jasje has quit
      • Shelly joined the channel
      • Shelly has quit
      • alastairp
        lucifer: from cron dumps log, "2023-02-03 00:00:02,800 listenbrainz.webserver ERROR Invalid dump ID 1191, could not find previous dump"
      • is this possibly related to the postgres master switchover, causing a jump in the PK sequence?
      • mayhem
        I hear MB is mentioned here: I haven't listened to it yet... https://open.spotify.com/episode/63lTbBpT0IJtJh...
      • Etua joined the channel
      • Etua has quit
      • alastairp
        I also heard that it was mentioned here: https://open.spotify.com/track/4cOdK2wGLETKBW3P...
      • lucifer
        alastairp: PK makes sense. about dumps, yeah its likely due to switchover.
      • }8]
        not sure if this is the right place but i have a physical cd that i was going to rip to flac. the album is in musicbrainz but the discid wasnt associated with it so i submitted that. the problem is now that the artists real name is used rather than his performance/artist name. whats the best/correct way to change the artist name for an album?
      • specifically, the album is "The Second Coming" by "Roger Troutman II" (his artist/performer name), but in musicbrainz its listed at "The Second Coming" by "Roger Troutman Jr." (his real name)
      • alastairp
        }8]: you should be able to get some feedback on this question in the #musicbrainz channel
      • }8]
        ok thanks
      • alastairp
        (while there is an overlap in the membership of both channels, this one tends to focus more on the software development of musicbrainz, while that one has more editors of the database)
      • }8]
        ahh ok, sorry, i didnt realize that
      • alastairp
        no problem at all!
      • jivte joined the channel
      • ArjunM joined the channel
      • Sum joined the channel
      • yvanzo
        Does BrainzBot has backups? If not, the data volumes of the new dockerized instance could be collected by borg backup.
      • alastairp: By deploying on the current Postgres instance, it seems that you just addressed OTHER-319, right?
      • BrainzBot
        OTHER-319: Update BrainzBot dependencies https://tickets.metabrainz.org/browse/OTHER-319
      • alastairp
        yvanzo: yes, this is in the middle of happening. By moving to MeB infra we have chatlogs (and botbot configuration) in the main PG cluster, which gives us automatic backups using the current backup system.
      • There is a redis server, but I don't think it's useful to back up its volume using borg, it's just used as a transient message queue between the chatbot and plugins
      • yvanzo
        👍
      • alastairp
        in terms of python dependency upgrades, I've already applied Leo's changes to upgrade to django 1.11 and python 3, with a pending task to upgrade to django 2.2. I've also started chopping a lot of code out of the original chatlots webapp from the original upstream project that we forked from (no longer maintained)
      • I think we've managed to remove approximately half of the dependencies, which makes ongoing maintenance much easier
      • unfortunately there is one large dependency used to generate bootstrap html from python which isn't supported with the latest version of django, so we can't yet upgrade to that
      • However we've been talking for years about how the web interface is terrible, and this may be a chance to rewrite the interface that generates the chat logs allowing us to make something simpler that doesn't have bad javascript, and at the same time reduce the number of external dependencies that the app has
      • reosarevok
        bitmap, yvanzo: hi! :)
      • alastairp: oh, something that doesn't suck would be nice
      • alastairp
        reosarevok: taking feedback for ideas
      • reosarevok
        "something that doesn't suck" is a good starting point ;D
      • alastairp
        but honestly, 1 URL per day with a prev/next button, and real working #anchors is probably 100x better than what we currently have
      • or even just static html per month like what we used to have :-P (that being said, I do like the search)
      • reosarevok
        +1 - only thing we'd need is the search
      • alastairp
        yeah, search is the one thing that makes a fully static system more difficult
      • personally I think the search works quite well - itd just be great if the results actually linked to the message :-P
      • petitminion joined the channel
      • yvanzo
        alastairp: Thanks, I’ve assigned the ticket to you to reflect the current status. Feel free to create sub-task if needed.
      • alastairp
        thanks yvanzo
      • bitmap
        reosarevok, yvanzo hello
      • yvanzo
        hi you both
      • The banner text is gone from the beta website, do you have it somewhere?
      • A tip is to comment banner text with <!-- --> for temporarily displaying a different message.
      • bitmap
        yes, just added it back
      • yvanzo
        Thank you
      • bitmap
        yesterday I spent well over and hour investigating those "Two relationships with the same key" errors and didn't get anywhere, so I'm working on improving the error message to give more information
      • yvanzo
        So there are 68 tickets addressed in beta, and 70 tickets reporting about beta.
      • reosarevok
        Seems like a good plan
      • yvanzo
        Which is really good to have that much feedback for beta :)
      • reosarevok
        There's also a bunch of useful comments on https://community.metabrainz.org/t/help-test-th...
      • (new ones)
      • https://community.metabrainz.org/t/help-test-th... for example is a funny issue, although not surprising
      • bitmap
        heh. yes, I saw some of those
      • yvanzo
        It’s getting wild, it would probably help to move it to tickets (existing or new ones)?
      • reosarevok
        The discussion being there is good (not everyone can be bothered to comment on tickets) but we should probably create tickets for some of these
      • bitmap
        tickets would help make tracking the issues a bit easier for me. I'm not sure how many of these will be fixed before monday, though
      • yvanzo
        It’s good to have feedback for sure, it’s better to have it in a structured way issues can be tracked, for example with... an issue tracker?
      • bitmap
        i'd prioritize the ones that crash the page or throw errors
      • reosarevok
        If we need a bit of time to try to debug the clashing rels error, maybe we should keep this in beta one more week anyway?
      • I can enter some tickets based on the feedback, at least
      • bitmap
        it's better to have an extra week than rush it, I'd say
      • yvanzo
        bitmap: Do you want us to start with helping on a specific issue you are stuck with? reviewing the existing tickets? or referencing the issues mentioned in this thread as tickets?
      • (Or any other way we can help with getting a more clear idea of what is left to be done or to be postponed)
      • bitmap
        I was mainly stuck on MBS-12860 and MBS-12861 yesterday but it seems like they are not easy to reproduce without more info. help with opening issues from comments in the thread would be helpful
      • BrainzBot
        MBS-12860: beta: Unable to enter relationships: "Two relationships with the same key" https://tickets.metabrainz.org/browse/MBS-12860
      • MBS-12861: Beta: "Error: Two relationships with the same key" does not specify the problematic relations https://tickets.metabrainz.org/browse/MBS-12861
      • reosarevok
        If you can fix 12861 (so, make the error give better info) then by having it in beta a bit longer we might get more info from users who hit it :)
      • bitmap
        yup. unfortunately these errors weren't logged to sentry and I'm not sure why
      • reosarevok
      • (happy to make changes on the second if needed)
      • bitmap
        we do have a captureExceptions call for any errors that occur
      • so either sentry was down, or blocked by both users, or the error capturing didn't work
      • reosarevok
        IIRC we've had the impression at least one of the users does block sentry before?
      • yvanzo
        Ok, I’m starting with “opening issues from comments in the thread”. I will mention each post here before opening a ticket, and edit the post to link to the ticket after.
      • reosarevok
        yvanzo: ok, you start from the top I start from the bottom? : )
      • yvanzo
        👍
      • bitmap
        one of them might but I've seen errors in sentry for the other before
      • reosarevok
        Hmm, then maybe there's a second issue with the logging, ok
      • "The never ending “show more”. Would be much better to show how many more there actually are in the list. “Show more (10 items)” or similar." bitmap: do you know where we show "show more"? https://beta.musicbrainz.org/artist/dc8c2046-46... already shows "See all 105 relationships", but maybe that's in the release rel editor?
      • duncan joined the channel
      • phw_ joined the channel