#metabrainz

aerozol: guidelines require a lot more community input :) but it might be a good time to start them!

If you have ideas, put them to paper (or well, to wiki) and we can open a discussion

Lotheric: goldenshimmer: not entirely sure but one possible lead I discussed with outsidecontext is the recently played endpoint returning tracks from spotify out of order. that can causes streams to miss from LB.

mayhem: alastairp: it looks like spotify endpoint might be returning streams out of order. for eg: consider a user continously listening since 1 PM. suppose we query at 1:15 PM and we get no streams. we attempt again later but still no streams then we query at 3 PM, endpoint returns streams till 2 PM + a recent stream listened around 2:50 PM. the endpoint hasn't yet returned streams between 2 PM - 2:50 PM. now we query the

endpoint again later, it has the entire listening history from 12- 3 at this point. but since we already imported the 2:50 PM stream in an older run, LB will ignore all the streams prior to 2:50 PM. so the user's listening history between 2-2:50 PM goes missing.

we don't have a reproducing sample currently but i, outsidecontext and people on spotify forums have seen similar behaviour https://community.spotify.com/t5/Spotify-for-De...

thoughts on how we can confirm this behaviour and then work on a workaround/fix.

I also try to monitor this today. Two days ago Spoitify import worked smoothly for me. Yesterday it had big issues, importing only 3 of 34 tracks (see https://gist.github.com/phw/03ce9343f64cb57328d...)

today there seem to be issues again. The three tracks I listened to earlier today are not yet showing up in Spotify API over an hour later. Maybe I can see it happening again and catch one of the bad API responses

[listenbrainz-server] 14amCap1712 opened pull request #1822 (03master…startup-bug): Fix LB server local development https://github.com/metabrainz/listenbrainz-serv...

treeshateorcs[m]: above chage should fix server startup error you were seeing yesterday. will merge after review but you can make the changes locally till then if you want.

wait ill adds scrobble to lfm or lb?

lucifer: btw i wanted to ask you something, wouldn't it be better to manually just add headers to the handle_error function, rather than using the decorator. Pretty sure the code quality would take a hit. What do you say

riksucks: yes i am currently debugging that error, if we can get it working fine otherwise adding manually is fine.

Right I see, thanks

riksucks: no useful leads yet, lets add the header manually.

jsonify returns a Response object so you can set the header on that.

lucifer: hmm, interesting. we use a limit to decide to no longer import some items from the spotify stream, right?

given that we have deduplication during ingest, maybe we just grab all 50 all the time?

(morning)

that would significantly increase dupes. also 50x the load on ts writer.

moooin!

we currently store the timestamp of the latest listen imported for a user from spotify and then query the api for streams only after that timestamp.

moooin!

how about we normally do an incremental (since last timestamp) fetch, but every 10 tries we do a full pull of all 50 listens?

interesting thought, that could work but also means increased load for 7mins every ~75mins.

ideally we would spread the larger checks out, so they don't bunch up.

I think we ought to build a whole class in the importer that sets the next check time for a given user.

and that class can contain a whole lot of logic or attempts at predicting things.

and perhaps that even has some things that could be tuned on the fly.

just to confirm - does the ts writer ever back up with work?

i was thinking of doing some redis based dedup but that would be more work to get right. store last 50 listens imported for the user in redis, then query all available spotify listens for the user, match against redis only send the ones not in redis to ts writer update redis.

currently no.

say that this "out of order mode" from spotify is an abberation. maybe we can turn the full checks off when things run normally?

lucifer: I think adding extra work on that level will just end up being a pain. let TS sort it out, but lets try to be smart about how often we query a particular user.

^ agreed about the complexity of another level

re disabling full checks, that needs us to be able to figure out when spotify is going out of order. for eg, outsidecontext noticed the issue but i usually don't monitor what spotify is importing for me so would never know.

makes sense.

yes. I also don't know how often it failed for me without noticing. Just yesterday I easily spotted it, and a few weeks ago there was another case I noticed.

lucifer: yes, perhaps this is something we check for and turn on automatically?

And today of course. Still waiting for my listenings from 3 hours ago to show up in Spotify API

like do random long pulls and check for OOO issues?

+1 on the "the importer that sets the next check time for a given user."

we found forum posts on the spotify site asking about this problem too, so it's not just related to us

we could have a button somewhere that says: "spotify is being dumb, please try harder for a day"

yes we can do random long pulls but still need to compare it with something to figure out that listens are missing.

there are two aspects to this: 1. spotify sometimes having huge delays until listens show up. There is obviously not much to do about this

and if 3 people press the button, we try harder.

is there a way for us to independently identify this without user input?

oh, ah. big brain time. 🧠

and 2. likely spotify sometimes only shown more recent listens first, while not yet showing some older ones. at least so far the theory, not yet seen an actual API result showing this behavior

randomly sample n users, get their listens every hour, see if listens "turn up" in the middle compared to when we checked last

lets get a paid spotify account. lets create a bot that listens to music. and it listens to a short track every 3 minutes or so.

now we have a "clock frequency" we know that listens should be coming for this user every 3 minutes.

if reality differs from theory, try harder.

sounds good

outsidecontext, have other listens for today showed up yet?

no, and I only did these three in the morning. I'll listen to something additional now for testing

outsidecontext, also i found another comment in the forums mentioning that it could be related to offline mode or iOS app. does that sound familiar to when you noticed the issue ?

have been using the desktop app on my laptops

👍

interesting is that the current playback endpoint is working well

yeah, I think we noticed that too

yeah the current playback endpoint usually works realtime but the recently played one may lag hours.

O’Moin

moin :)

yvanzo: someone said "nothing in the MusicBrainz documentation about work types, like Incidental Music and such. the only place I’ve found those descriptions is on the Create New Work pages. frankly, what we’ve got hasn’t helped me figure out if parts of a soundtrack are incidental or not"

This goes together with the ws ticket you shared yesterday

Should we look into having, say, /work-types on the site, and then maybe a /ws/2/work-types JSON representation of the same list?

reosarevok: a friend of mine, with whom I am chatting about a possible collaboration, asks:

"In the meantime can you send me a link to a musicbrainz entry that you think gives a good representation of full metadata? Like an artist with ISRCS and other unique identifiers, etc. So I can get a vibe on what the ultimate goal for artists would be."

got any artists in mind that fit this bill especially well?

Hmm. Artists specifically

Lemme think

or an album...

where the artists is also pretty well pimped out.

Well, as much as I dislike his bullshit, https://musicbrainz.org/artist/164f0d73-1234-4e... seems well-filled. Also, wtf, does the US really allow one to legally change their name to just "Ye", no surname? :D

For a classical example, https://musicbrainz.org/artist/ae0b2424-d4c5-4c... should work fine

(I see four recordings were added during the holidays that need fixing, heh, will change them)

Is the possible collaboration a seekrit?

ISRCs and whatnot though depend a ton on what people have sent and I can't guarantee most recordings for either artist have them, tbh

Since those usually require CD in hand and even then it's not always there

Kanye has a fair amount for albums, it seems, but not many for singles

ok, thanks for those links.

what would we consider ideal minimal metadata for an artist who wishes to add one release, or even perhaps one single track they just finished?

artist name, sortname, type, area, link to their own page, a link to where the artists can be supported, metadata for release including a link where it can be obtained.

is kinda my thinking.

Sort name can be confusing, but yes. Link to their own page (which often will mean social media), link to streaming pages or whatnot, track titles and durations

And ideally ISRCs since it's much easier for them to provide them than for anyone else to find them

IPI and ISNI in the form confuse a lot of people though, so if we have a section for ISRCs it should specify "it's ok to skip them if you don't have them"

"but if there is a way to make an MB add-on for these" <- there is fierce competition for doing this right now. and each of these tools is collecting data for their own "WE MUST WIN IT ALL OR WE PERISH" approach. which is.. uhm, not going to work.

But they are publishing music on several digital streaming platforms at the same time. Might it be possible to add MB as a target? It would just grab metadata and not audio content.

anything is possible at this point in time.

it would be good for us to think about this in broad terms of what we would love to see happen.

But you’re probably right that they may not allow easily to create add-on for their software.

and then Marc and I can actually see about what is possible.

yvanzo: and the add-ons are usually for larger studios with established artists. byta deals more with tons of teeny artists.

which is actually great -- we want more teeny artists metadata in MB.

Right.

Yeah, established artist data we'll eventually get most of anyway

the easiest would be a cd stub like system, but... that doesn't do a lot of good in the grand scheme of things.

having switched on artists would be even better.

I mean, it does do good if we or byta have people in charge of finishing the import

But it doesn't otherwise

agreed.

but I have no ideal of the costs and scalability of this.

Yeah, me neither :) Something like that would need to be tested for feasibility for sure

But the scalability issue will to some degree be there even if the artists do all the adding since the new additions will be seen by the community

My main worry with artists doing the adding is just "who deals with edit notes by the community"

The artist is likely gone by then

agreed.scalability remains the greatest concern.

perhaps we can fund one position of someone who is what you call a finisher.

Of course, byta edits could be clearly be marked as such, with a note like "this was added in this way, if you see something wrong here, be bold and make the changes, if you see something consistently wrong, get in touch so we can improve the system"

if we can identify the steam of users coming in from this, this person could be tasked with tidying up this incoming stream of data.

But basically there should be clarity in any case on whether the community can expect an answer to notes or not

Yeah. That seems perfectly doable

(I do that at a small scale rn with BBC works)

The scalability is the main question

You'd need to decide with byta whether each artist would get one account, or all would go through one main byta account

a good question, that.

If they have accounts for artists on their site, then giving them an automatic [byta_or_whatever_prefix]_username account is probably doable (needs implementing, but)

an ideally, this would be less of "marc and I deciding" but more of "reo telling us how it could be feasible"

If they don't and each time the artist adds new music they do it on a form without them having an account over there as such, then a general account for them seems simpler

the "creating an account" seems to be a significant hurdle for such services.

Yeah. We could in some way automate MB account creation for them if they have an account on their side

so, if they are required to have a byta account, then adding yet another one is a likely dealbreaker.

I wasn't thinking of the artist filling in our captcha :)

Just that behind the scenes they'd get assigned an account

once we migrate to single oauth on MeB, then this is much easier.

For submitting the data

agreed, not a bad idea.

But that only works if they have byta accounts already

If not, they should go through one generic byta account

That'd probably be easier to monitor anyway, but harder to find the issues with specific artists and support them if they need to

FWIW though we now have an edit search for edit note content, meaning different accounts wouldn't be a problem if we can just search for a string such as "Submitted via byta"

One generic byta account seems to be more workable indeed.

yvanzo: thanks for the weblate email.