#metabrainz

/

      • lucifer[m]
        mayhem: nothing much to be found in the logs.
      • 2025-05-16 13645, 2025

      • lucifer[m]
        zas: can we have same monitoring for timescale on gaga as we have the main pg cluster?
      • 2025-05-16 13639, 2025

      • lucifer[m]
        mayhem, zas: also i think it would be nice to upgrade the LB database. MB/LB etc already upgraded last year. possibly do a server upgrade too while at it?
      • 2025-05-16 13645, 2025

      • _BrainzGit
        [listenbrainz-server] 14granth23 opened pull request #3279 (03master…Linting-Issues): Fixed Linting Issues in Stats related files https://github.com/metabrainz/listenbrainz-server…
      • 2025-05-16 13619, 2025

      • holycow23[m] joined the channel
      • 2025-05-16 13619, 2025

      • holycow23[m]
        Hey lucifer, sorry for the delay — I was caught up with exams. I’ll be getting back on track starting today. Just wanted to check if you’re done with the dump, so I can begin setting things up and then start with the stats work.
      • 2025-05-16 13647, 2025

      • lucifer[m]
        holycow23: should be done sometime next week, i have tested the code and produced a sample dump and imported it locally. the overall process works, currently on refactoring the overall framework before integrating sample dumps it.
      • 2025-05-16 13656, 2025

      • lucifer[m]
        remind me what metadata do you need?
      • 2025-05-16 13605, 2025

      • holycow23[m]
        Thanks for the update — sounds good!... (full message at <https://matrix.chatbrainz.org/_matrix/media/v3/download/chatbrainz.org/jTgClzhHxyOzelbYAeKWmidR>)
      • 2025-05-16 13614, 2025

      • holycow23[m]
        * Thanks for the update — sounds good!... (full message at <https://matrix.chatbrainz.org/_matrix/media/v3/download/chatbrainz.org/uMOeJAodEKZAsRmgFwqPPxyl>)
      • 2025-05-16 13634, 2025

      • reosarevok[m]
        Lotheric: I think it's IA?
      • 2025-05-16 13650, 2025

      • mshahnawaz1[m] has quit
      • 2025-05-16 13608, 2025

      • reosarevok[m]
        Ok, having the bootlegs sort properly by date again on test.mb is satisfying
      • 2025-05-16 13645, 2025

      • HemangMishra[m] joined the channel
      • 2025-05-16 13645, 2025

      • HemangMishra[m]
        Hi aerozol: , please review the mockups for the new onboarding flow in lb android.
      • 2025-05-16 13645, 2025

      • HemangMishra[m]
      • 2025-05-16 13625, 2025

      • reosarevok[m]
        yvanzo, bitmap tested things around in test.mb, the stuff I could think of seems to work fine. I'm not sure how to test the pdf covers since uploading images to test does not seem to be allowed anymore (we used to do that...)
      • 2025-05-16 13654, 2025

      • yvanzo[m]
        Hi lucifer: Where did you keep track of SIR performance tests? I’m testing full reindex at the moment and it’s taking eons.
      • 2025-05-16 13646, 2025

      • yvanzo[m]
        I’m not sure if any changes to the config file were needed.
      • 2025-05-16 13647, 2025

      • zas[m]
        lucifer: which timescale monitoring?
      • 2025-05-16 13616, 2025

      • lucifer[m]
        yvanzo: i and bitmap ran it independently on mb docker and the full reindex always finished in 12-13 hours.
      • 2025-05-16 13648, 2025

      • yvanzo[m]
        Thank you, that is about the same time here. I thought that we improved it, never mind. :)
      • 2025-05-16 13608, 2025

      • lucifer[m]
        zas: i mean the main db cluster has a monitoring dashboard like this: https://stats.metabrainz.org/d/000000067/postgres…. but the timescale db instance on gaga doesn't.
      • 2025-05-16 13607, 2025

      • zas[m]
        Ah yes, that's pg, I think it requires a special user on pg side (ask bitmap), it depends on telegraf settings like -> https://github.com/metabrainz/metabrainz-ansible/…
      • 2025-05-16 13632, 2025

      • aerozol[m] has quit
      • 2025-05-16 13620, 2025

      • _BrainzGit
        [listenbrainz-server] 14MonkeyDo merged pull request #3279 (03master…Linting-Issues): Fixed Linting Issues in Stats related files https://github.com/metabrainz/listenbrainz-server…
      • 2025-05-16 13641, 2025

      • _BrainzGit
        [bookbrainz-site] 14MonkeyDo merged pull request #1172 (03master…dependabot/npm_and_yarn/babel/runtime-7.26.10): chore(deps): bump @babel/runtime from 7.23.2 to 7.26.10 https://github.com/metabrainz/bookbrainz-site/pul…
      • 2025-05-16 13654, 2025

      • adhawkins has quit
      • 2025-05-16 13658, 2025

      • _BrainzGit
        [bookbrainz-site] 14MonkeyDo merged pull request #1173 (03master…dependabot/npm_and_yarn/babel/helpers-7.27.1): chore(deps): bump @babel/helpers from 7.20.7 to 7.27.1 https://github.com/metabrainz/bookbrainz-site/pul…
      • 2025-05-16 13607, 2025

      • adhawkins_ joined the channel
      • 2025-05-16 13615, 2025

      • adhawkins_ is now known as adhawkins
      • 2025-05-16 13615, 2025

      • mayhem[m]
        lucifer: you want to upgrade PG on kiss? We just added a new drive to kiss, it feels pretty weird now getting rid of it.
      • 2025-05-16 13623, 2025

      • mayhem[m]
        what would you go for instead?
      • 2025-05-16 13638, 2025

      • mayhem[m]
        zas: Do we have a spare slot for this orchestration?
      • 2025-05-16 13630, 2025

      • zas[m]
        We have 2 spare slots (reserved)
      • 2025-05-16 13619, 2025

      • mayhem[m]
        nice.
      • 2025-05-16 13630, 2025

      • lucifer[m]
        mayhem: the database is on gaga so we wouldn't be getting rid of kiss for now.
      • 2025-05-16 13602, 2025

      • mayhem[m]
        oh right, we run both on one. and gaga is a bit older, no?
      • 2025-05-16 13656, 2025

      • lucifer[m]
        sorry, not sure what you mean by both on one?
      • 2025-05-16 13613, 2025

      • mayhem[m]
        both timescale and normal pg are running on gaga.
      • 2025-05-16 13619, 2025

      • lucifer[m]
        the normal pg runs on main cluster with MB and other projects' DB.
      • 2025-05-16 13626, 2025

      • lucifer[m]
        only timesclae runs on gaga.
      • 2025-05-16 13612, 2025

      • lucifer[m]
        we do use timescale db for storing playlists/mapping/similarity/metadata caches in normal pg tables though.
      • 2025-05-16 13612, 2025

      • mayhem[m]
        ok, rather than making more bad guesses, assumptions...
      • 2025-05-16 13618, 2025

      • mayhem[m]
        which machine would you like to upgrade?
      • 2025-05-16 13606, 2025

      • lucifer[m]
        atj: proposed it in the past and i think it would make sense to combine the two databases into one. but that would mean additional backups setup.
      • 2025-05-16 13610, 2025

      • lucifer[m]
        i would like to upgrade gaga.
      • 2025-05-16 13635, 2025

      • mayhem[m]
        gaga is $107/month. 128GB ram. 16 cores. 3.5TB in two drives.
      • 2025-05-16 13646, 2025

      • Maxr1998 has quit
      • 2025-05-16 13630, 2025

      • mayhem[m]
      • 2025-05-16 13646, 2025

      • mayhem[m]
        there isn't a clear replacement in view.
      • 2025-05-16 13659, 2025

      • mayhem[m]
        if we change servers, it looks like it will be more expensive for the same.
      • 2025-05-16 13610, 2025

      • mayhem[m]
        sanity check me, since my brain is not working well today.
      • 2025-05-16 13610, 2025

      • zas[m]
        AX102 is likely faster than current gaga
      • 2025-05-16 13653, 2025

      • julian45[m] has quit
      • 2025-05-16 13657, 2025

      • mayhem[m]
        and more expensive to get to 3.5TB of Raid-0 storage.
      • 2025-05-16 13614, 2025

      • Maxr1998 joined the channel
      • 2025-05-16 13628, 2025

      • mayhem[m]
        yeah, nothing good is in sight.
      • 2025-05-16 13655, 2025

      • lucifer[m]
        Cool I'll check if there is a worthwhile upgrade option there otherwise will just upgrade the timescale db and not the machine.
      • 2025-05-16 13611, 2025

      • mayhem[m]
        yeah, seems like the best plan.
      • 2025-05-16 13641, 2025

      • zas[m]
        mayhem: AX102 + 1 x 3.84 TB SATA SSD Datacenter Edition = 147€/month
      • 2025-05-16 13653, 2025

      • zas[m]
        What is the disk space used for? pg?
      • 2025-05-16 13608, 2025

      • mayhem[m]
        timescale/pg, yes.
      • 2025-05-16 13652, 2025

      • mayhem[m]
        zas[m]: I dont see that one. Current gaga is only $107.
      • 2025-05-16 13610, 2025

      • zas[m]
        The SATA SSD is an option (+43€)
      • 2025-05-16 13656, 2025

      • zas[m]
        actually that's 152€ at Falkenstein
      • 2025-05-16 13621, 2025

      • zas[m] uploaded an image: (120KiB) < https://matrix.chatbrainz.org/_matrix/media/v3/download/matrix.org/pWddfQVtqkoHWiNsMmPTEnLS/image.png >
      • 2025-05-16 13624, 2025

      • mayhem[m]
        but then our storage is not redundant.
      • 2025-05-16 13635, 2025

      • zas[m]
        it isn't in RAID0 either
      • 2025-05-16 13658, 2025

      • mayhem[m]
        clearly we need RAID-1.
      • 2025-05-16 13616, 2025

      • mayhem[m]
        * 3.5TB of <del>Raid-0, * Raid-0</del>, * raid-1 storage.
      • 2025-05-16 13635, 2025

      • mayhem[m] fixed the previous mesg.
      • 2025-05-16 13620, 2025

      • zas[m] uploaded an image: (80KiB) < https://matrix.chatbrainz.org/_matrix/media/v3/download/matrix.org/zAAIaRumKmuanwfohOSBbneS/image.png >
      • 2025-05-16 13621, 2025

      • zas[m]
        Ok, then we go for 4 NVME in RAID10 ->
      • 2025-05-16 13622, 2025

      • zas[m]
        It will be faster for pg, has enough disk space, and redundancy
      • 2025-05-16 13629, 2025

      • mayhem[m]
        50€ more for some... CPU speedup we don't really need.
      • 2025-05-16 13638, 2025

      • zas[m]
        Then we go for AX52, cpu is already faster than gaga's one, but it's much cheaper
      • 2025-05-16 13638, 2025

      • zas[m]
        it has only 64gb ram by default though, so we can increase it
      • 2025-05-16 13651, 2025

      • zas[m]
        grrr, doesn't work, no big NVME drives :(
      • 2025-05-16 13613, 2025

      • mayhem[m] uploaded an image: (35KiB) < https://matrix.chatbrainz.org/_matrix/media/v3/download/chatbrainz.org/rtVjEWVbtrYqhqZAwLEwknsX/image.png >
      • 2025-05-16 13621, 2025

      • \- joined the channel
      • 2025-05-16 13655, 2025

      • \- has quit
      • 2025-05-16 13604, 2025

      • \- joined the channel
      • 2025-05-16 13653, 2025

      • \- has quit
      • 2025-05-16 13652, 2025

      • _BrainzGit
        [listenbrainz-server] 14MonkeyDo merged pull request #3271 (03master…Persistence-player-volume): FIX:LB-1793 Persistence volume https://github.com/metabrainz/listenbrainz-server…
      • 2025-05-16 13653, 2025

      • BrainzBot
        LB-1793: BrainzPlayer: Save Volume % between sessions https://tickets.metabrainz.org/browse/LB-1793
      • 2025-05-16 13646, 2025

      • Rachel[m] has quit
      • 2025-05-16 13618, 2025

      • lucifer[m]
        mayhem: zas: okay i'll just upgrade the database for now.
      • 2025-05-16 13613, 2025

      • Sophist-UK has quit
      • 2025-05-16 13638, 2025

      • _BrainzGit
        [sir] 14yvanzo merged pull request #142 (03master…2to3): SEARCH-594: Port Sir to Python 3 https://github.com/metabrainz/sir/pull/142
      • 2025-05-16 13639, 2025

      • BrainzBot
        SEARCH-594: Port Search Index Rebuilder (SIR) to Python 3 https://tickets.metabrainz.org/browse/SEARCH-594
      • 2025-05-16 13636, 2025

      • lucifer[m]
        🎉🎉
      • 2025-05-16 13616, 2025

      • lucifer[m]
        almost ~2.5 years since first commit.
      • 2025-05-16 13651, 2025

      • _BrainzGit
        [sir] 14yvanzo closed pull request #109 (03master…master): Port Sir to Py3 SEARCH-594 https://github.com/metabrainz/sir/pull/109
      • 2025-05-16 13652, 2025

      • BrainzBot
        SEARCH-594: Port Search Index Rebuilder (SIR) to Python 3 https://tickets.metabrainz.org/browse/SEARCH-594
      • 2025-05-16 13652, 2025

      • yvanzo[m]
        almost ~6 years since ticket creation.
      • 2025-05-16 13617, 2025

      • reosarevok[m]
        Still a lot faster than some MBS tickets!
      • 2025-05-16 13626, 2025

      • reosarevok[m]
        Good job everyone :)
      • 2025-05-16 13631, 2025

      • yvanzo[m]
        Not really.
      • 2025-05-16 13632, 2025

      • reosarevok[m]
        Not really good job? Well, hopefully good enough that it works :)
      • 2025-05-16 13601, 2025

      • Goemon
        better than never!
      • 2025-05-16 13629, 2025

      • reosarevok[m]
        yvanzo, lucifer do we still keep using mbdata from git on sir then or did v30 get pushed to pypi in the meantime?
      • 2025-05-16 13607, 2025

      • yvanzo[m]
        We keep using it from git as it has not been released to PyPI yet.
      • 2025-05-16 13614, 2025

      • lucifer[m]
        reosarevok: i haven't fixed it to push to pypi yet so let's keep git.
      • 2025-05-16 13624, 2025

      • reosarevok[m]
        Ok, perfect
      • 2025-05-16 13630, 2025

      • yvanzo[m]
        The answer to your question is just here: https://pypi.org/project/mbdata/#history
      • 2025-05-16 13622, 2025

      • yvanzo[m]
        reosarevok: Please rebase your SIR PRs if you have time. :)
      • 2025-05-16 13626, 2025

      • reosarevok[m]
        Rebased https://github.com/metabrainz/sir/pull/160 then on the new master
      • 2025-05-16 13628, 2025

      • reosarevok[m]
        hah
      • 2025-05-16 13654, 2025

      • reosarevok[m]
        🥷
      • 2025-05-16 13611, 2025

      • yvanzo[m]
        Even the AC MBIDs’ one if you don’t mind.
      • 2025-05-16 13640, 2025

      • reosarevok[m]
        Can do!
      • 2025-05-16 13636, 2025

      • reosarevok[m]
        Hmm https://ci.metabrainz.org/job/sir/830/console - does the test image need to be updated somehow as well for schema change?
      • 2025-05-16 13638, 2025

      • yvanzo[m]
        reosarevok: yes
      • 2025-05-16 13656, 2025

      • yvanzo[m]
        Well, it should be built while running the tests.
      • 2025-05-16 13629, 2025

      • reosarevok[m]
        yvanzo: rebased https://github.com/metabrainz/sir/pull/157 - should it point to normal master?
      • 2025-05-16 13649, 2025

      • reosarevok[m]
        (since rn it is confusing with all the extra stuff)
      • 2025-05-16 13638, 2025

      • reosarevok[m]
        I cherry-picked the same commit as on my own PR for version updates since yours seemed to be intermediate testing ones anyway, hope that's fine
      • 2025-05-16 13614, 2025

      • yvanzo[m]
        reosarevok: either way is fine
      • 2025-05-16 13635, 2025

      • reosarevok[m]
        yvanzo: I see there's a new `test` image for `metabrainz/musicbrainz-test-database`, should we use that in `docker-compose.test.yml` in `sir`? Should we make a new image `schema-change-2025-q2` or something?
      • 2025-05-16 13644, 2025

      • reosarevok[m]
        (or is this a red herring)
      • 2025-05-16 13606, 2025

      • yvanzo[m]
        reosarevok: `test` has been pushed 1 day ago, according to https://hub.docker.com/r/metabrainz/musicbrainz-t…
      • 2025-05-16 13637, 2025

      • yvanzo[m]
        bitmap: I guess that is a schema 30 database? ^
      • 2025-05-16 13614, 2025

      • reosarevok[m]
        It seems to be
      • 2025-05-16 13632, 2025

      • reosarevok[m]
        I just dunno if we want to have it point to test or we want a more permanent version
      • 2025-05-16 13659, 2025

      • yvanzo[m]
        we can keep that for now
      • 2025-05-16 13619, 2025

      • reosarevok[m]
        Ok, I'll push a commit to use that to the PRs then
      • 2025-05-16 13609, 2025

      • julian45[m] joined the channel
      • 2025-05-16 13610, 2025

      • julian45[m]
        zas: following up on SSO testing request from a few days ago; if you have a sec, please try going to outline.meb.o and seeing if you can log in
      • 2025-05-16 13602, 2025

      • yvanzo[m]
        reosarevok: Oh, do you mean instead of `production`, right?
      • 2025-05-16 13606, 2025

      • reosarevok[m]
        Yes
      • 2025-05-16 13649, 2025

      • yvanzo[m]
        Yes it is fine for the PRs, thank you.
      • 2025-05-16 13608, 2025

      • yvanzo[m]
        Not sure if the currently tagged image is supposed to be the final one.
      • 2025-05-16 13632, 2025

      • yvanzo[m]
        bitmap: Would it be fine or premature to tag `test` as `schema-change-2025-q2` for `musicbrainz-test-database`?
      • 2025-05-16 13651, 2025

      • reosarevok[m]
        Ok, I actually fixed the tests in https://github.com/metabrainz/sir/pull/160 now too which I had entirely skipped originally 🫣
      • 2025-05-16 13638, 2025

      • reosarevok[m]
        I guess the same needs to happen on https://github.com/metabrainz/sir/pull/157 since you also didn't touch the test files, will look into it
      • 2025-05-16 13653, 2025

      • bitmap[m]
        <yvanzo[m]> "bitmap: Would it be fine or..." <- it should be fine, I don't think anything will change in the image now
      • 2025-05-16 13623, 2025

      • bitmap[m]
        though we can probably merge schema-change-2025-q2 into master now
      • 2025-05-16 13606, 2025

      • zas[m]
        julian45: I can
      • 2025-05-16 13653, 2025

      • julian45[m]
        great, thanks
      • 2025-05-16 13627, 2025

      • julian45[m]
        that means we've confirmed functionality for both SAML and OIDC-based SSO, which is great
      • 2025-05-16 13612, 2025

      • monkey[m] has quit
      • 2025-05-16 13627, 2025

      • bitmap[m]
        <zas[m]> "Ah yes, that's pg, I think it..." <- we just have a regular `telegraf` user, AFAICT it doesn't have any special attributes or privileges (cc lucifer)
      • 2025-05-16 13653, 2025

      • reosarevok[m]
        bitmap: merging into master seems fine to me tooo
      • 2025-05-16 13620, 2025

      • bitmap[m]
        done
      • 2025-05-16 13649, 2025

      • reosarevok[m]
        Can you then make a new master image? Then I can drop the commits changing it to test
      • 2025-05-16 13635, 2025

      • bitmap[m]
        for musicbrainz-test-database right? you just have to build docker images, I started one at https://ci.metabrainz.org/job/musicbrainz-docker-…
      • 2025-05-16 13631, 2025

      • reosarevok[m]
        Thanks
      • 2025-05-16 13649, 2025

      • reosarevok[m]
        Oh wait, duh, the previous tag was production, not master
      • 2025-05-16 13601, 2025

      • reosarevok[m]
        So I still need a commit