#metabrainz

mayhem: nothing much to be found in the logs.

zas: can we have same monitoring for timescale on gaga as we have the main pg cluster?

mayhem, zas: also i think it would be nice to upgrade the LB database. MB/LB etc already upgraded last year. possibly do a server upgrade too while at it?

[listenbrainz-server] 14granth23 opened pull request #3279 (03master…Linting-Issues): Fixed Linting Issues in Stats related files https://github.com/metabrainz/listenbrainz-serv...

Hey lucifer, sorry for the delay — I was caught up with exams. I’ll be getting back on track starting today. Just wanted to check if you’re done with the dump, so I can begin setting things up and then start with the stats work.

holycow23: should be done sometime next week, i have tested the code and produced a sample dump and imported it locally. the overall process works, currently on refactoring the overall framework before integrating sample dumps it.

remind me what metadata do you need?

Thanks for the update — sounds good!... (full message at <https://matrix.chatbrainz.org/_matrix/media/v3/...>)

* Thanks for the update — sounds good!... (full message at <https://matrix.chatbrainz.org/_matrix/media/v3/...>)

Lotheric: I think it's IA?

Ok, having the bootlegs sort properly by date again on test.mb is satisfying

Hi aerozol: , please review the mockups for the new onboarding flow in lb android.

yvanzo, bitmap tested things around in test.mb, the stuff I could think of seems to work fine. I'm not sure how to test the pdf covers since uploading images to test does not seem to be allowed anymore (we used to do that...)

Hi lucifer: Where did you keep track of SIR performance tests? I’m testing full reindex at the moment and it’s taking eons.

I’m not sure if any changes to the config file were needed.

lucifer: which timescale monitoring?

yvanzo: i and bitmap ran it independently on mb docker and the full reindex always finished in 12-13 hours.

Thank you, that is about the same time here. I thought that we improved it, never mind. :)

zas: i mean the main db cluster has a monitoring dashboard like this: https://stats.metabrainz.org/d/000000067/postgr.... but the timescale db instance on gaga doesn't.

Ah yes, that's pg, I think it requires a special user on pg side (ask bitmap), it depends on telegraf settings like -> https://github.com/metabrainz/metabrainz-ansibl...

[listenbrainz-server] 14MonkeyDo merged pull request #3279 (03master…Linting-Issues): Fixed Linting Issues in Stats related files https://github.com/metabrainz/listenbrainz-serv...

[bookbrainz-site] 14MonkeyDo merged pull request #1172 (03master…dependabot/npm_and_yarn/babel/runtime-7.26.10): chore(deps): bump @babel/runtime from 7.23.2 to 7.26.10 https://github.com/metabrainz/bookbrainz-site/p...

[bookbrainz-site] 14MonkeyDo merged pull request #1173 (03master…dependabot/npm_and_yarn/babel/helpers-7.27.1): chore(deps): bump @babel/helpers from 7.20.7 to 7.27.1 https://github.com/metabrainz/bookbrainz-site/p...

lucifer: you want to upgrade PG on kiss? We just added a new drive to kiss, it feels pretty weird now getting rid of it.

what would you go for instead?

zas: Do we have a spare slot for this orchestration?

We have 2 spare slots (reserved)

nice.

mayhem: the database is on gaga so we wouldn't be getting rid of kiss for now.

oh right, we run both on one. and gaga is a bit older, no?

sorry, not sure what you mean by both on one?

both timescale and normal pg are running on gaga.

the normal pg runs on main cluster with MB and other projects' DB.

only timesclae runs on gaga.

we do use timescale db for storing playlists/mapping/similarity/metadata caches in normal pg tables though.

ok, rather than making more bad guesses, assumptions...

which machine would you like to upgrade?

atj: proposed it in the past and i think it would make sense to combine the two databases into one. but that would mean additional backups setup.

i would like to upgrade gaga.

gaga is $107/month. 128GB ram. 16 cores. 3.5TB in two drives.

there isn't a clear replacement in view.

if we change servers, it looks like it will be more expensive for the same.

sanity check me, since my brain is not working well today.

AX102 is likely faster than current gaga

and more expensive to get to 3.5TB of Raid-0 storage.

yeah, nothing good is in sight.

Cool I'll check if there is a worthwhile upgrade option there otherwise will just upgrade the timescale db and not the machine.

yeah, seems like the best plan.

mayhem: AX102 + 1 x 3.84 TB SATA SSD Datacenter Edition = 147€/month

What is the disk space used for? pg?

timescale/pg, yes.

zas[m]: I dont see that one. Current gaga is only $107.

The SATA SSD is an option (+43€)

actually that's 152€ at Falkenstein

but then our storage is not redundant.

it isn't in RAID0 either

clearly we need RAID-1.

* 3.5TB of <del>Raid-0, * Raid-0</del>, * raid-1 storage.

Ok, then we go for 4 NVME in RAID10 ->

It will be faster for pg, has enough disk space, and redundancy

50€ more for some... CPU speedup we don't really need.

Then we go for AX52, cpu is already faster than gaga's one, but it's much cheaper

it has only 64gb ram by default though, so we can increase it

grrr, doesn't work, no big NVME drives :(

[listenbrainz-server] 14MonkeyDo merged pull request #3271 (03master…Persistence-player-volume): FIX:LB-1793 Persistence volume https://github.com/metabrainz/listenbrainz-serv...

LB-1793: BrainzPlayer: Save Volume % between sessions https://tickets.metabrainz.org/browse/LB-1793

mayhem: zas: okay i'll just upgrade the database for now.

[sir] 14yvanzo merged pull request #142 (03master…2to3): SEARCH-594: Port Sir to Python 3 https://github.com/metabrainz/sir/pull/142

SEARCH-594: Port Search Index Rebuilder (SIR) to Python 3 https://tickets.metabrainz.org/browse/SEARCH-594

🎉🎉

almost ~2.5 years since first commit.

[sir] 14yvanzo closed pull request #109 (03master…master): Port Sir to Py3 SEARCH-594 https://github.com/metabrainz/sir/pull/109

SEARCH-594: Port Search Index Rebuilder (SIR) to Python 3 https://tickets.metabrainz.org/browse/SEARCH-594

almost ~6 years since ticket creation.

Still a lot faster than some MBS tickets!

Good job everyone :)

Not really.

Not really good job? Well, hopefully good enough that it works :)

better than never!

yvanzo, lucifer do we still keep using mbdata from git on sir then or did v30 get pushed to pypi in the meantime?

We keep using it from git as it has not been released to PyPI yet.

reosarevok: i haven't fixed it to push to pypi yet so let's keep git.

Ok, perfect

The answer to your question is just here: https://pypi.org/project/mbdata/#history

reosarevok: Please rebase your SIR PRs if you have time. :)

Rebased https://github.com/metabrainz/sir/pull/160 then on the new master

hah

🥷

Even the AC MBIDs’ one if you don’t mind.

Can do!

Hmm https://ci.metabrainz.org/job/sir/830/console - does the test image need to be updated somehow as well for schema change?

reosarevok: yes

Well, it should be built while running the tests.

yvanzo: rebased https://github.com/metabrainz/sir/pull/157 - should it point to normal master?

(since rn it is confusing with all the extra stuff)

I cherry-picked the same commit as on my own PR for version updates since yours seemed to be intermediate testing ones anyway, hope that's fine

reosarevok: either way is fine

yvanzo: I see there's a new `test` image for `metabrainz/musicbrainz-test-database`, should we use that in `docker-compose.test.yml` in `sir`? Should we make a new image `schema-change-2025-q2` or something?

(or is this a red herring)

reosarevok: `test` has been pushed 1 day ago, according to https://hub.docker.com/r/metabrainz/musicbrainz...

bitmap: I guess that is a schema 30 database? ^

It seems to be

I just dunno if we want to have it point to test or we want a more permanent version

we can keep that for now

Ok, I'll push a commit to use that to the PRs then

zas: following up on SSO testing request from a few days ago; if you have a sec, please try going to outline.meb.o and seeing if you can log in

reosarevok: Oh, do you mean instead of `production`, right?

Yes

Yes it is fine for the PRs, thank you.

Not sure if the currently tagged image is supposed to be the final one.

bitmap: Would it be fine or premature to tag `test` as `schema-change-2025-q2` for `musicbrainz-test-database`?

Ok, I actually fixed the tests in https://github.com/metabrainz/sir/pull/160 now too which I had entirely skipped originally 🫣

I guess the same needs to happen on https://github.com/metabrainz/sir/pull/157 since you also didn't touch the test files, will look into it

<yvanzo[m]> "bitmap: Would it be fine or..." <- it should be fine, I don't think anything will change in the image now

though we can probably merge schema-change-2025-q2 into master now

julian45: I can

great, thanks

that means we've confirmed functionality for both SAML and OIDC-based SSO, which is great

<zas[m]> "Ah yes, that's pg, I think it..." <- we just have a regular `telegraf` user, AFAICT it doesn't have any special attributes or privileges (cc lucifer)

bitmap: merging into master seems fine to me tooo

done

Can you then make a new master image? Then I can drop the commits changing it to test

for musicbrainz-test-database right? you just have to build docker images, I started one at https://ci.metabrainz.org/job/musicbrainz-docke...

Thanks

Oh wait, duh, the previous tag was production, not master

So I still need a commit