#metabrainz

/

      • BrainzGit
        [listenbrainz-android] 1407jasjeet merged pull request #524 (03main…fix_player_pager): Feat: fixed the player screen and mini player pager https://github.com/metabrainz/listenbrainz-andr...
      • mayhem[m]
        lucifer: it could really use more docstrings, but a lot of these classes are derived so they are not really needed.
      • but for base classes like UserEntity, please add docstrings and then for the derived classes add a note about looking at the base class for docs. how does that sound?
      • lucifer[m]
        sure sounds good.
      • BrainzGit
        [listenbrainz-android] 1407jasjeet merged pull request #528 (03main…Fixed-AppCrash): Update Bottom NavigationBar with NullSafety https://github.com/metabrainz/listenbrainz-andr...
      • mayhem[m]
        lucifer: I *finally* got past the "stale DB handle" problem in mb_metadata_cache.
      • the whole process would finish and the with .... : statement that opened the DB connection would, I guess, fail to cleanup the collection and then threw an error. when nothing really went wrong. lame.
      • so now I can setting a success flag and if that is set, then ignore the error. its finally moved on to building the artist data -- first time in months from what I can see.
      • zas[m]
        mayhem: julian45 the report might be for metabrainz/docker-rsyncd container running on aretha
      • yvanzo[m]
        bitmap, lucifer, reosarevok: ping 🏓
      • mayhem[m]
        ohhhhhh. I didn't consider that.
      • julian45[m]
        zas[m]: good point, i will poke at that when i get a second
      • lucifer[m]
        mayhem: ahaan i see, makes sense.
      • yvanzo: hi!
      • reosarevok[m]
        Hi!
      • mayhem[m]
        zas: metabrainz.org is down for some reason. I restarted the container, just in case. no errors, but the page doesn't load.
      • do you see anything odd at the gateways?
      • reosarevok[m]
        I see bitmap is not around yet it seems, but he put up https://github.com/metabrainz/musicbrainz-docke... yesterday
      • (I haven't managed to test it yet because docker)
      • mayhem[m]
        zas: seems like a consul issue -- the metabrainz-prod container does not have a valid config.py
      • LB is down too. MB is up.
      • zas[m]
        I'll have a look
      • mayhem[m]
        ah, LB responded finally. could be a DB issue?
      • I've seen DB errors in the logs.
      • lucifer: can you please help look what is wrong? MeB is down, LB was down.
      • lucifer[m]
        mayhem: hmm container seems to be up fine.
      • maybe a gateways issue?
      • mayhem[m]
        the metabrainz-prod container has no config.py
      • thus it can't start correctly.
      • lucifer[m]
        it has consul_config.py
      • it started fine because i can see uwsgi started workers.
      • mayhem[m]
        hmm, ok. so yes, the container is up ok.
      • zas: any signs from the gateways?
      • reosarevok[m]
        yvanzo: I put a new commit up also to make grafana depend on sql-exporter, which hopefully makes it come up when you do it your way
      • yvanzo[m]
        reosarevok: I'm confused, why "because docker"?
      • lucifer[m]
        zas, mayhem : MB db is rejecting connections.
      • causing pages to hang.
      • mayhem[m]
        MB or MeB?
      • reosarevok[m]
        yvanzo: Well, because the step that bitmap specified doesn't work for me and says the file doesn't exist and my docker knowledge is not there yet :)
      • lucifer[m]
        sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) server closed the connection unexpectedly
      • This probably means the server terminated abnormally
      • before or while processing the request.
      • MB.
      • reosarevok[m]
        (to figure out wtf)
      • lucifer[m]
        at least for LB.
      • mayhem[m]
        sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) server closed the connection unexpectedly... (full message at <https://matrix.chatbrainz.org/_matrix/media/v3/...>)
      • in the LB error log. I think this is the same slow query that monkey flagged recently.
      • I see timescale is happy, PG for LB is not.
      • reosarevok[m]
        yvanzo: I was waiting for bitmap to point where I'm being dumb and then I will test it :)
      • mayhem[m]
        bitmap: are you around?
      • lucifer[m]
        mayhem[m]: this query is in MB db.
      • reosarevok[m]
        No he is not, we were supposed to be meeting him but :)
      • He'll appear
      • lucifer[m]
        sorry MeB db.
      • mayhem[m]
        yep.
      • lucifer: could it be related to my long running cron job for updating all mb caches?
      • lucifer[m]
        not sure i couldn't find anything in PG logs
      • mayhem[m]
        been running for a couple of hours now. I wonder if that has backed things up.
      • BrainzGit
        [listenbrainz-server] 14Aerozol merged pull request #3135 (03master…data-interval-doc): Update data-update-intervals.rst https://github.com/metabrainz/listenbrainz-serv...
      • mayhem[m]
        15 minutes left on that job.
      • let me kill it just in case.
      • killed.
      • mayhem[m] sobs
      • BrainzGit
        [listenbrainz-android] 1407jasjeet opened pull request #529 (03main…main): Bug fixes https://github.com/metabrainz/listenbrainz-andr...
      • [listenbrainz-android] 1407jasjeet merged pull request #529 (03main…main): Bug fixes https://github.com/metabrainz/listenbrainz-andr...
      • mayhem[m]
        nope that wasn't it.
      • jasje[m]
        mayhem: lucifer please ping me when the server is up and stable, I'm planning to make a release. Want things working so another release doesn't get rejected.
      • mayhem[m]
        MB is up and working totally fine. But other projects connecting to the DB are not fine. Did the PG accounts for the other projects get borked?
      • jasje[m]: we'd like that too!
      • monkey[m]
        BookBrainz seems to be loading fine. One difference is that it only connects to the BB database (so DB server OK, MB DB having issues?)
      • mayhem[m]
        MB loads fine too!
      • it seems the non-MB accounts in PG are not happy.
      • monkey[m]
        Wouldn't that affect BB too?
      • mayhem[m]
        does BB use MB?
      • monkey[m]
        Not the MB DB, but it uses the same PG instance, with its own account
      • mayhem[m]
        understood.
      • those appear to be working fine, but from what I can gather, the non mb account are not working right.
      • reosarevok[m]
        What does CB use?
      • Because that seems fine
      • mayhem[m]
        unsure.
      • monkey[m]
        MB and BB databases
      • Not sure WRT PG accounts, but I assume its own account
      • mayhem[m] posts to bsky
      • bitmap[m]
        <mayhem[m]> "bitmap: are you around?" <- I'm here
      • mayhem[m]
        good morning!
      • could you please help with diagnosing a problem
      • yvanzo[m]
        Hi bitmap: Sorry, I have to go soon.
      • mayhem[m]
        LB and MeB are currently down.
      • it seems that connections to the MB DB are being rejected.
      • CB and BB are fine. MB is totally fine.
      • bitmap[m]
        sorry I'm late yvanzo
      • yvanzo[m]
        reosarevok: Please report issues to all of us (bitmap, lucifer and I) when you find these, without waiting for the next meeting, or we will hardly make any progress.
      • bitmap[m]
        are they connecting to jimmy or hendrix?
      • mayhem[m]
        bitmap[m]: unsure, let me dig
      • lucifer[m]
        should be connecting to current master
      • mayhem[m]
        10.2.2.25:65436
      • hendrix
      • yvanzo[m]
        bitmap: No worries, thanks for the new PR.
      • Let’s postpone the meeting to another time, when there are no ongoing maintenance issues.
      • reosarevok[m]
        Ok, I can look into testing the PR in the meantime - should we try again tomorrow?
      • bitmap[m]
        the only weird thing I see in jimmy's logs rn are things like... (full message at <https://matrix.chatbrainz.org/_matrix/media/v3/...>)
      • I'm not sure what that indicates yet though. I don't see anything related to LB
      • mayhem[m]
        connecting to the db using musicbrainz_ro connects just fine.
      • lucifer[m]
        are there any pgbouncer lgos to see if its blocking connections to meb db?
      • zas[m]
        rsyncd issue on aretha is fixed
      • bitmap[m]
        cl_active and sv_active are very high for the metabrainz db right now
      • mayhem[m]
        @zas can you please help on the LB/MeB issue right now?
      • bitmap[m]
        what server is meb hosted on?
      • mayhem[m]
        two sites down, no clue what is going on.
      • hendrix
      • bitmap[m]: hendrix
      • lucifer[m]
        what's sl_active and cl_active?
      • mayhem[m]
        lots of DB errors in the logs, but connecting the DB via psql connects fine.
      • zas[m]
        I see a lot of messages in syslog like:
      • Jan 22 16:51:36 hendrix consul[33496]: 2025-01-22T16:51:36.572Z [INFO] agent.http: Request cancelled: method=GET url="/v1/health/service/pgbouncer-aretha?index=16073260&passing=1&stale=&wait=60000ms" from=172.17.0.9:41036 error="context canceled"
      • Jan 22 16:51:37 hendrix consul[33496]: 2025-01-22T16:51:37.609Z [INFO] agent.http: Request cancelled: method=GET url="/v1/health/service/pgbouncer-aretha?index=16073260&passing=1&stale=&wait=60000ms" from=172.17.0.9:41134 error="context canceled"
      • bitmap[m]
        lucifer[m]: the number of server/client connections to pgbouncer
      • lucifer[m]
        mayhem: just to clarify, LB web is down not LB api.
      • mayhem[m]
        yes, indeed.
      • bitmap[m] sent a code block: https://matrix.chatbrainz.org/_matrix/media/v3/download/chatbrainz.org/wzAihjGAnMvMRaZWlhpmbsEG
      • bitmap[m]: I had a long running script that *might* have set things off.
      • SELECT editor_id... (full message at <https://matrix.chatbrainz.org/_matrix/media/v3/...>)
      • lucifer: this query has been flagged as slow -- its possible that that backed things up and when I ran the cron job it overloaded PG bouncer.
      • lucifer[m]
        mayhem: yes i am creating an index to try speed it up
      • mayhem[m]
        fabulous/
      • lucifer[m]
        -------------------------------------------------------------------------------------------------------------------------------------------------... (full message at <https://matrix.chatbrainz.org/_matrix/media/v3/...>)
      • seems to have worked.
      • mayhem[m]
        very good.
      • zas[m]
        I restarted metabrainz web container, and it works
      • bitmap[m]
        I increase the pool sizes I bit
      • lucifer[m]
        i restarted lb web as well
      • bitmap[m]
        s/increase/increased/, s/I/a/
      • mayhem[m]
        logs much quieter now
      • bitmap[m]
        40+15 reserver for both metabrainz_db and listenbrainz (up from 20/10)
      • mayhem[m]
        MeB is back.
      • bitmap[m]
        s/reserver/reserve/, s/metabrainz_db/metabrainz\_db/
      • lucifer[m]
        i think it will take some time to pile back up if there is another issue.
      • will keep an eye on it for an hour or so.
      • mayhem[m]
        yay, thanks @bitmap and @lucifer for sorting this.
      • monkey: that slow flairs query you flagged is now taken care of. I can't find the sentry report for it again -- can you please mark is resolved?
      • monkey[m]
        Sure thing
      • Actually, link in the MEB ticket
      • Resolving
      • dvirtz[m] joined the channel
      • dvirtz[m]
        Hi. Can I DM anyone regarding development server setup?
      • mayhem[m]
        we prefer that you ask the question here.
      • dvirtz[m]
        Alright. Should the sample database have all the attributes? That's what I mainly need for developlment
      • mayhem[m]
        Which database for which project? MB? Which attributes?
      • vardhan__ has quit