#metabrainz

hey bitmap you around?

mayhem: when you have time, can you please add https://github.com/metabrainz/pylistenbrainz/pu... to coreteam

yvanzo: see MBS-13343 - where should docker issues be reported? Github issues rather than jira? If so we should indicate it somewhere in jira maybe, if possible

MBS-13343: musicbrainz docker stuck on port 5000 even with env variable set to something else https://tickets.metabrainz.org/browse/MBS-13343

moooin!

lucifer: done!

Hmm. Seems babel is struggling with flow all of a sudden? for gettext:

yvanzo: ^ seen that one before?

feels nice to not be wasting *huge* amounts of resources here. https://ludic.mataroa.blog/blog/what-the-goddam...

yvanzo, bitmap: is there any difference between "delete" and "remove"?

"This place has no relationships and will be removed automatically in the next few days. If this is not intended, please add more data to this place." - it probably should say "add some relationships" rather than just "more data", which is what we already say for works?

If not (since we have "more data" elsewhere where the data can be releases or recordings for example) then we should probably change the work one to be consistent

reosarevok: not in musicbrainz AFAIK. "delete" sounds a bit more permanent, I guess

Pratha-Fish: hey, I'm back

Yeah, wondering because AFAICT we use them weirdly interchangeably

So maybe we should stick to one

(it also still annoys me that some entities have an /add url and some a /create url)

so delete remove or remove delete?

not sure which one we use more

relete demove?

We use remove a lot more, so I'd probably drop delete

Or well, it feels like we do, "Remove {entity}" edits et

*etc

Seems we mostly use deleted for editors

But not only

I guess keeping it just for that meaning could make sense, if you say it sounds more permanent?

bitmap https://usercontent.irccloud-cdn.com/file/tHlCx...

I think I prefer remove too

I'll get back to you in ~10 minutes right after I finish lunch. Having some issues comparing areas from the two sources 🫠

ok :)

hey reosarevok if you have 10 minutes, we can also maybe take up this opportunity to do an overall survey of the whole project?

I do have 10 minutes, yes :) So we can do that

The current issue that I am facing is as follows:

We have pristine data coming from both sources, in almost exactly the same format.

e.g.

(generating, just a sec)

NVM

Trying to demonstrate the issue has solved it? :)

Figured it out? :D

kellnerd: I only wish 😭

Ran into a little bug

Anyway

The structure of the data fetched from musicbrainz is almost the same as well. But the id columns in it use musicbrainz_ids instad of wikidata ids

Soo basically comparing based on ids is not an option

You should be comparing based on the wikidata ids you need to get from the URLs :)

The big idea is, to somehow index areas even with the same names but different subdivisions and countries such that each area generates a unique index for it to be queried

reosarevok: that's exactly what I am trying right now!

Not sure how well it's gonna work tho, but let's hope that ALL entries in musicbrainz have a wikidata id associated with it

*id -> url

99.99% should

As for the second issue, we have some repeating areas going with the same subdivision and country, but different wikidata URLs

lemme fetch an example real quick

Ahh where are teh examples when you need them 🫠

Here we go: https://www.wikidata.org/wiki/Q61962594 https://www.wikidata.org/wiki/Q6917624

The big idea with these entries is detecting them.

That does seem to be a full duplicate, yeah

They're probably rare, but I'm sure there will be more

I was just wondering what could we even do about it

I think it seems fine to skip those / log them

If you detect you'd add a second area with the same name and parents at least, you could log it and not do it automatically

I can calculate them based on the city name, but what if they turn out to be cities with same name but different subdivision? or even country?

So I tried generating an index of such areas to filter out such areas, but here's the result

(since if we *do* add it we should actually add a disambiguation anyway so it's better if a human does it)

We have a lot of city data with no parent areas 💀 https://usercontent.irccloud-cdn.com/file/c5qDc...

Hmm. In MB?

Or in WD?

Its from MB

In MB cities with no parents should be rare

but same story from wiki

Actually wiki could be suffering from some other issue due to my own bad code

But currently facing it https://usercontent.irccloud-cdn.com/file/zaqO0...

reosarevok: let's see how rare

wikidata having a few amount of those seems more likely, you should just ignore them if so

a few amount. me fail english that's unpossible

reosarevok: we have 4885 of those in MB surprisingly

That does not sound right

Examples? :)

on it

Apparently there's also something wrong with my data 😑

Will have to take a moment to get it fixed

That's ok

Ah yes.

I had to reset my dev environment a couple of days ago

So looks like I am missing something in the musicbrainz database

Here's how I've set it up so far:

1. Dropped musicbrainz_db and reimported it using the datadumps generated by bitmap (contained all areas and relations last I checked.)

2. Created the materialized tables as well

But before the reset, my query used to fetch around 700k rows along with parent data. But now it barely fetches 200k with NO PARENT DATA

you sure you imported the second dump and not the first one?

Yes, I am pretty sure

But I'll try again just in case

select count(*) from area_containment; -> 320019 in production

Yep, looks like I don't have area_containment data 🤦‍♂️

I remember running the materialized table command in musicbrainz docker as well.

I'll try restarting it

is the table completely empty?

There was no table detected 💀

lol

it should always exist in the schema, even if it's empty

lmao

are you sure you spelled the table name right?

[musicbrainz-server] 14reosarevok opened pull request #3066 (03master…use-delete-for-user-only): Use "remove" consistently for non-editor cases https://github.com/metabrainz/musicbrainz-serve...

pretty sure

I'll try dropping and reimporting the dump then ig

It takes a while tho

But while it's done, let's discuss the current state of the project ig

where is the table "not detected"? how are you checking for it?

*being executed

could also be a search_path issue

rant this in bash `psql -U musicbrainz -h localhost -c "select count(*) from area_containment;" -d musicbrainz`

is your database name musicbrainz or musicbrainz_db? (you mentioned the latter earlier, but your psql command uses musicbrainz)

🤦‍♂️

fixed it and it worked 💀

Yay

the count is 320019

😁

Thankgod. Those resets take quite a while on my device 💀

Yayy now my query is fetching parent areas as well!

Yay

Give me a sec, I am running some compute to figure out fatherless areas on musicbrainz

Perfect

No fatherland for you!

bitmap: "Automatically subscribe me to artists I create", "Batch-create new works", "This user has not created any entities"

All those sound like it should be "added" to me ...

"Only the editor who created an edit can cancel it." that sounds like it should be entered :)

Alright, looks like we still have 2349 fatherless cities in musicbrainz 🫠

Ok, examples :)

reosarevok: both your suggestions sgtm

I think I might look over "create" strings next then

Pratha-Fish: I'm around for 15 min more max, and then in a few hours btw :)

Alright, trying to make it as fast as possible

It's fine later too :)

But let's see

Check the GID, open it in MB, see if it seems correct

aerozol: Saw these fun CSS gradients, thought of you: https://codepen.io/thebabydino/details/GRRpzNX

reosarevok Here's the whole dump https://usercontent.irccloud-cdn.com/file/OWJ2f...