#musicbrainz-devel

damnit, why has no one done this work for me!

warp: go ahead

ocharles: great :D

the systemd timer stuff is annoying for things like logrotate on laptops that don't have long uptimes - if you have the timer set to every 12h starting from boot time and never leave it running for 12h, the timer will never trigger.

so cron is still useful.

winter :(

it feels as if it's 20:00 but it isn't even 18:00 yet.

apparently the DST change is this weekend here.

kepstin-work: I suppose another option is to be super cloudy, and use one process that is fired on the hour to find open edits and send messages to a message queue

and another process that waits on the message queue

dunno if that'll make it better or worse :)

then you can have modbot farms if we suddenly need to process millions of edits an hour

:P

also I feel old and grumpy just for noticing + complaining about it :)

warp: it's 4:50pm here and dark outside :(

ocharles: heh, but that would act strangely if the queue gets backed up - edits will be in the queue multiple times.

i suppose just making the queue drop messages that can't be processed would be ok, tho.

kepstin-work: depending on how you queue is built

what's wrong with closing edits as they expire?

you can have a queue that has a concept of unique ids

luks: nothing, that's what this modbot does

luks: implementing that is hard, because edits don't tell you when they expire

yeah, but I think most poeple find it confusing

but if you mean the second a user enters a vote, then I wouldn't want that, because that loads the web server

luks: so you have to check all the edits to see if they have expired yet

kepstin-work: if you have database triggers and a message queue, you can know exactly when they expire

ocharles: web server could send it to the expired edits queue :)

oh, true

luks: did you end up using pg_amqp for anything in the end?

I haven't used it for anything but hobby stuff myself

nope

luks: basically, expiring edits immediately means having a process that knows when all the upcoming edit expirations are going to be, and just sleeps until the next one before firing a notification of some sort - it should be doable, i think...

expiring edits immediately just needs a queue, a modbot, and something pushing messages when something happens that expires an edit

as was said before

fsvo "immediately", anyway

there could be still some "grace period"

but it would be fixed amount of time

not waiting to the next hour

sounds like something to discuss :)

no gaming the system if it always expires an edit exactly 1h after the 3rd yes vote, eh :)

does anybody know what exactly is the plan with the "ingestr"?

luks: there isn't an exact plan yet.

as I understand it, to be something that can take arbiratry data and index it, such that people can later view the data and say "this path is the artist name and this path is the release name", so please open that in the release editor

Why is it spelled all stupid-like?

the first step is to expose data and make it searchable.

and create very simple import steps

hawke_: a play on flickr

ocharles: O RLY?

then to let the community give us feedback on if its useful and how to make it better.

so since we have IA data, the point is to use that to expose it.

and then we need to work on matching tools for matching foreign data sets to MB.

luks: I saw that you offered to do some matching work for Brewster.

do you have a plan for that yet?

what kinds of input formats would ingestr be looking at supporting? just sutff like xml? pluggable frontends?

ruaok: I'm running the matching script already

luks: cool.

is the source somewhere where we can play with it and use it for other data sets?

which is related to my question, I have albums which probably match, but I'd like somebody to review it

and I'm not sure if a web app for that conflicts with the ingestr somehow

luks: yep, that is the next step. we need to find a way to manage that

for more than one data set.

ruaok: https://github.com/lalinsky/musicbrainz-matchin...

but I'm working on it as it runs

and I honestly dont have a good way laid out in my mind on how to do that.

thanks!

I'm doing fuzzy matching based on track lengths and then validating track titles to make sure I don't have false positive

great. I think that approach makes sense.

so I'm thinking of creating a simple app that displays a random album from that category and asks the user to verify it

there isn't that many of such matches there, but it's more than I can handle personally :)

I like that.

this sounds somewhat similar to what the matching code in picard does.

I think there is a possibility of lots of such matches. not all data sets incoming will be as clean at what we're getting from the archive.

I've been thinking about indexing discogs as well and if I don't find a match in MB at all, but I do find a match in discogs, offer the user to import it from discogs

luks: wouldn't that auto-match a release to its remaster or something?

*maybe* even import it automatically from discogs

luks: yep.

reosarevok: yes, probably

luks: I think if we make the matching *REALLY* conservative, and people review the results then maybe we can do just that.

reosarevok: but if I want to deal with that, I can forget about this kind of matching

I imagine that's not in the interests of the IA - I imagine they'd want a copy of both original and remaster and to know which one is which

But I don't know if there's anything in their data that can allow us to know

luks: there are lots of people literally throwing data sets at us.

(say barcodes or catnos or something)

and we really need a comprehensive solution for dealing with them. exposing the data, importing clean data and all that.

oh, and I think we can also harvest data from these data sets.

reosarevok: people generally do not keep that kind of information in tags

such as picking barcodes from these data sets.

luks: maybe we should store matched data in ingrestr too. (this record matches to MBID blah)

then we can pluck extra data out automatically via a bot.

luks: obviously for the ones they digitise themselves, that's basic stuff to include

So I trust the IA will include it. But for the rest, yes, dunno

the only way to get that kind of information, if it's there, is parsing the textual descriptions

and I would love ingrestr to be in python. :(

IA's indexing is pretty primitive regarding metadata

maybe we should start over with a more clear goal in mind. :)

ruaok: the goal seems clear? "manage different sets of metadata, automatically find matches between them and with MusicBrainz, and provide users an easy way to confirm them or import the ones that do not match"

What seems hard is the execution :p

that's too broad goal, IMO

for ingestr, yes.

each data set will be different

there needs to be a web app component too.

luks: that's the point

luks: our plan is to keep mappings between different data sets in ingrestr.

Isn't the idea to turn each of the data sets into something compatible with all the rest?

reosarevok: that is my idea, yes.

I see this working as a two step process.

1. ingest data and show unstructured results.

2. let the community look at it and figure out a mapping for the new data.

3. install the mapping

4. import data.

making different data sets "compatible" is not going to work

(and you get two extra steps for free!)

- he says and lists 4 steps.

:p

luks: how is it not going to work?

that's huge amount of work, in cases it's even possible

luks: I dont think that is the goal.

I mean, basically it's mapping each set to MB's approach

(which makes them compatible-through-MB)

in different data sets you have different information and you can use that information primarily for matching to MB

I view it as picking matching data components out.

sometimes you don't have that, but you have something else, that the other data set doesn't

But when you know what matches where, you can also look for places where the datasets match in their MB-matching

and making it easier to import, but in a lot of cases direct import wont work well.

And be like "huh, these might be the same thing!"

reosarevok: what's the point of keeping the data after you match them to MB?

luks: to send that data back to the source

luks: for the ones which match to MB, dunno

was one argument, anyway

I care about the ones which do *not*

(say, we get info from two different sources but the release has the same cat# or EAN or track times)

it might also be useful for saying: this data is BAD. don't want.

prevent repeat import attempts

Precisely because, as you said, different sets bring different info, it's useful to try to put it all together

(same as I, manually, might search site X's data to find a barcode and then use that barcode to find a release on amazon to find a back cover to find more data - just bottily :p)

I don't think it will every cherry-pick data like that

*ever

Well, that's the obvious use of having multiple datasets coming in

So it would be a bit sad not to try to take advantage of it

realistically, you will be happy if you get album/title/artist

which is not even good for automatic import

heh

I guess we're used to different data sources :p

Of course, someone would have to ask them for the data, but I imagine that might actually work

I haven't seen label data for pop so I don't know how much that sucks