#metabrainz

[musicbrainz-server] 14mwiencek merged pull request #3183 (03schema-change-2024-q2…MBS-13421): MBS-13421: Add support for genre collections https://github.com/metabrainz/musicbrainz-serve...

aerozol: currently the person has to reinstall the app to fix such an issue. But we could make a logout button in the next release

aerozol: about blog post, sure. just write me. just not sure how ready this is for any blog post (spoiler: there will be no ETA)

akshaaatt: thanks, have passed that on

outsidecontext: great, I’ll think of some questions! No ETA’s, 100% agreed

yvanzo: ping

monkey, mayhem: around for discussing roadmap?

sorta, but still kinda foggy in the head. could we do tomorrow, plz?

sure

thx

zas: hi! is there an empty vm or can you create a new one that i can use to test some spark stuff?

that I can help with.

I know zas just a review of things, so I suspect that there are none freely laying around.

I can create one. what specs do you need?

thanks!

any decent vm (laptop specs) would do for now

how long do you need it for?

a week probably

ok, I am calling the server deleteme-20240601 :)

65.109.165.197

thanks!

let me know if you can get in.

I dont think it installed my own key, lol.

nope, asks for a password

clearly I should not be working today.

oh. yes.

<= dumbass

ok, working now.

remember, lucifer this is a brand new VM, so your account does not exist.

log in as root

mayhem: trying as root, still asks for a password

I used the ssh key for kiss, if that makes a difference

i see, i logged into kiss and then tried to ssh into vm and that works.

i am in now. will add my ssh key as well.

thanks!

np.

and please please pleas ping me when you're done with it.

we keep forgetting them.

which is $$$

yes, will do!

Can do tomorrow too then for the roadmap chat

reosarevok: ping back

Hi! :) Just a reminder to check the two open PRs by bitmap in the schema change milestone if you have any time

I'm working on the read-only connector one some

I probably won't have time to work on MBS-13360, but we'll see

MBS-13360: Detect which indexes need to be rebuilt for collations https://tickets.metabrainz.org/browse/MBS-13360

reosarevok: I have been able to reproduce the cpanfile issue on wolf.

Glad to know it's not just me. But hope it's not too hard to solve :/

btw, I fixed a couple small issues I found via CheckSchemaMigration.sh:

Oh, duh, thanks for catching the dumb-dumb on the label code one

There's no reason the second drop (ALTER TABLE label DROP CONSTRAINT IF EXISTS label_code_length;) would need to apply to a mirror tho is there?

yeah, it shouldn't be needed, but can't hurt either

"Make sure DB_SCHEMA_SEQUENCE is set to 28 in lib/DBDefs.pm" that's a lot of schema changes. We're old!

that script is always run before the other one anyway

bitmap: so should I drop MBS-13360 from the fix version?

MBS-13360: Detect which indexes need to be rebuilt for collations https://tickets.metabrainz.org/browse/MBS-13360

I guess there's no reason that *must* happen at schema change time anyway?

it's happening all the time

yes, we can add running RebuildIndexesUsingCollations.pl to the upgrade notes anyway

Ok, dropped for now

Can always re-add if you find time for it

reosarevok: "happen" I was thinking of the bug though.

yeah I guessed :)

Did you have any luck with figuring out why?

No I mean the bug related to collation.

Aaaaah

I guessed wrong :D

But we can still add it ASAP after the schema change and it should be fine, right?

I'm preparing the schema change blog post but after that I could test things as needed

For whatever is useful to you two :)

No that would be another schema change, what bitmap suggested is to keep adding a manual step to upgrade notes instead if he doesn't find the time for it.

Hmm, ok. I thought these were schema change adjacent tickets but not schema changes themselves, because I didn't see it in the schema bit of https://blog.metabrainz.org/2024/03/12/schema-c...

But I see it's just not on any lists there :)

Maybe I got it wrong too.

I wasn't planning to implement MBS-13360 as any kind of schema change, rather a script that can run when the container starts

MBS-13360: Detect which indexes need to be rebuilt for collations https://tickets.metabrainz.org/browse/MBS-13360

It depends on the implementation actually.

^

so it could be added in a later release

👍

Then it sounds good to me to do just that and not add extra stress to the pre-schema change days :)

yvanzo: for the docker upgrades, what do we need to document? I see the psql 12 upgrade had its own separate script (as per https://github.com/metabrainz/musicbrainz-docke...)

bitmap: similar question, but for non-docker. https://blog.metabrainz.org/2020/05/18/postgres... has a lot of stuff we might or might not need this time.

We don't seem to have the equivalent to "The main steps you need to follow in this case are running the 20200518-pg12-before-upgrade.sql and 20200518-pg12-after-upgrade.sql scripts in that order" ?

that's almost the same situation, plus the solr data has to be rebuilt from scratch

At least https://github.com/metabrainz/musicbrainz-serve... doesn't seem to add any such scripts

correct, we don't have any additional scripts this time

So just install packages and pg_upgradecluster ?

do you mean one-off scripts?

yeah, the *before-upgrade.sql and *after-upgrade.sql scripts from the v12 one

we needed those because the pg_upgrade wouldn't work unless we dropped some things first

bitmap: The Docker issue seems to be that `carton` is run from another directory than `musicbrainz-server`. Did you encounter this before? `PERL_CARTON_PATH` seems to be for installed modules, not for `cpanfile`.

yvanzo: I'm not sure I ever attempted to run it outside of the musicbrainz-server dir, tbh

Here is the breaking change: https://github.com/metabrainz/musicbrainz-docke...

bitmap: Ok, so we just need to give the full path of dump files instead of running from their directory, right?

yvanzo: yeah, I'm not sure there's a way to override that

bitmap: at what part of the schema upgrade sequence should we upgrade psql? At the start? I guess it does matter since some new code uses psql16 stuff

So maybe between

Turn off cron jobs if you’re automatically updating the database via cron jobs.

and

Switch to the new code with git fetch origin followed by git checkout v-2024-05-13-schema-change.

?

sounds right

bitmap: apt-get update && apt-get install postgresql-16 postgresql-server-dev-16 should still work like for 12 last time or has anything changed? I see the PGDG repo seems to still work and have 16

yvanzo: apparently a path to the cpanfile can be specified with PERL_CARTON_CPANFILE

(similarly, do we still need pg_dropcluster --stop 16 main)

the (newer) version of carton we use isn't on metacpan for some reason

Thanks, I already modified the script to use full path for dump files though.

I couldn’t file any documentation mentioning this variable either.

yeah, it's only in the changelog for v0.9.63

reosarevok: I haven't checked but have no reason to think it still wouldn't automatically create a default cluster

the package names should be the same

we have v1.0.35

ah, it does have v1.0.35 at https://metacpan.org/dist/Carton

I was looking at dist/carton (lowercase) 😒

apparently I can only access the lowercase one over http, it loads Carton over https o_O

Thanks, I reverted my changes and used `PERL_CARTON_CPANFILE` as it can be useful in other situations too.

reosarevok: That should work for you too this time ^ (and I tested running `createdb.sh` again)

aerozol: hi! can you download and try to install https://drive.google.com/file/d/1UrzWIFUhUYkdtf... ?

yvanzo: ok, testing :) Thanks!

yvanzo: still getting the same, I guess I need to do docker compose build --no-cache --progress=plain musicbrainz then?

Will try

just building should be enough

(no need of --no-cache)

Ok :)

I was thinking that too :)