#metabrainz

Alright theflash__ (IRC) let me know if you need help

Hey aerozol , I was offline yesterday, Ill go through the replies today... thanks

np!

<yvanzo[m]> "It would be tricky to implement..." <- yvanzo: bitmap https://github.com/yellowHatpro/mb-exurl-ia-ser.... (full message at <https://matrix.chatbrainz.org/_matrix/media/v3/...>)

[listenbrainz-server] 14rimma-kubanova opened pull request #2910 (03master…applemusic): Importing playlists from Apple Music https://github.com/metabrainz/listenbrainz-serv...

lucifer: hi! I pushed the PR for Apple Music, please lemme know when you can review it, i will fix the errors

also monkey may I ask you to review the front side?

PR: https://github.com/metabrainz/listenbrainz-serv...

yvanzo: can you check sir queues? We still have many entries in search failed, and it keeps growing. Not sure how to interpret this.

aerozol: re playlists, I would first like to complete the core features of the user pages first and then do the playlists tab

aerozol: Could u lemme know what is to be done when user is already followed? Just need color and text of that button

reosarevok: hi! I wanted to ask if you can clarify some things regarding spotify lookups ( I remember you mentioned that we can ask you for help regarding MusicBrainz if needed.)

I’ve checked the spotify_lookup_from_mbid api and it seems it works by looking for an exact track match using the MBID to fetch the Spotify ID. Is that correct? Does that mean that each recording in MB contains spotify id?

I was thinking how can I implement the same thing for apple music for example

Hi! That wasn't written by the MusicBrainz team I think so I'd have to check, can you point me to the code?

lucifer: please let me know when you're about, I'd love a quick catchup regarding spotify playlists and stats issues.

<reosarevok[m]> "Hi! That wasn't written by the..." <- I was looking here: https://github.com/metabrainz/listenbrainz-serv...

If you're not sure, no worries at all! I guess it's outside of MB

Hi zas, on it.

yvanzo: zas can one of you catch me up to the current status of the solr cluster? is there anything I can do to help it become more stable?

mayhem, zas : SIR is currently indexing documents at full speed again.

have we switched to the new cluster yet?

rimskii: I think mayhem is the person for this :)

sorry for not keeping up, but last week sleep was utter hell. I feel somewhat normal today.

reosarevok[m]: okay thanks ! :)

Sorry for bothering you

mayhem: Not yet, we want to do it these days if not today. But we might have to keep the old cluster in parallel for a month or so.

ok, that's good. so all the problems, in theory, will go away soon? 🤞

rimskii: let me see if I can help you. we do not have 1:1 mapping of MBIDs to Spotify IDs.

(we wish, but alas no)

<rimskii[m]> "reosarevok: hi! I wanted to..." <- > <@kubrimskii:matrix.org> reosarevok: hi! I wanted to ask if you can clarify some things regarding spotify lookups ( I remember you mentioned that we can ask you for help regarding MusicBrainz if needed.)... (full message at <https://matrix.chatbrainz.org/_matrix/media/v3/...>)

yes, sure.

i say you already answered haha

s/say/saw/

the goal is to match a MBID to an apple music id, yes?

Yes

Just like for Spotify

lets step through this and see if we can work it out.

the first two "with" statements do:

1. Resolve redirects: MBIDs can at times be deprecated and point to newer MBIDs. This step ensures that the given MBIDs are currently valid MBIDs (e.g. they are valid or were redirected)

2. Resolve recording MBIDs to "canonical recording" MBIDs. A recording of a song may have more than one actual recording in MB, so this step ensures that we find the recording we consider to be "canonical". Canonical is important because we can find the metadata for each canonical recording easily in the metadata cache.

so in the for loop, we check the redirects, then the canonicals, then fetch the metadata.

all this was basically to take an MBID and resolve it to the correct metadata for that MBID.

because that is how the spotify data index is created.

then, finally the look up is done via the lookup_using_metadata(ordered_metadata) function.

any questions so far?

nope

good good.

maybe one

<mayhem[m]> "the first two "with" statements..." <- well If im going to do such lookup for apple music should I do the same here too?

so in the utils module I just linked, it attempts to find the right spotify matches. and note that it does not do that with spotify ids, but with metadata lookups.

I think in the end you'll pretty much copy everything that was done for spotify to the apple setup.

haha ok

ok

if I remember correcly lucifer said we do have for apple music

but not for soundcloud

that jives with my memory.

you can check the schema of the tables here: https://github.com/metabrainz/listenbrainz-serv...

on the surface, they seem very similar.

so far it seems that all the code between apple and spotify is wholly separate, so copying the spotify code and modifying it for apple seems to be the right approach.

i see

So as far as I got, the spotify_lookup first finds the right mbid then searches for spotify metadata, right?

yes

mayhem[m]: spotify metadata is spotify_cache here, right?

yes

ahh i see

I got it

thank you soo much, appreciate it much !! :)

no problem.

mayhem[m]: hi!

rimskii[m]: will check and let you know in a bit.

mayhem[m], outsidecontext: spark cluster was unavailable last week for a few days because of memory and storage issues apparently. i reset-ed the cluster on the weekend and did a fresh install, its processing requests as we speak. stats are done, recommendations and playlists queued.

mayhem[m]: the deleteme hetzner can be removed now.

i have setup the ansible playbooks for the spark cluster and used it to do the fresh installs.

Lucifer: one request -- when you expect downtime for the cluster, ping me please. So I can tweet/post about doing maintenance.

mayhem[m]: it was unexpected.

i discovered it when diagnosing why stats were not updating.

after discovering it, i got to fixing it but that ended up taking over a day.

Ah, another case for more monitoring. But next time you do notice, please let me know

yes sure

Deleteme server, deleted. Thanks.

(to be clear, i discovered the memory and storage issues first and then reset it, not the other way around)

btw we should be able to add new servers to the cluster with ease now.

should be handy for trying out MLHD data processing stuff.

Oh yisss! Great news!

shall we do that after south or even before?

s/south/oauth/

i guess after might be better

have to also deal with LB gaga disks and typesense

Jigen, bitmap[m]: restored the data. thanks!

volumetrique: yes, we currently don't consider track_mbids during mapping.

but iirc mayhem[m] and i discussed previously to change that (not to consider track mbids in mapping but prefer track mbids if submitted over mapping assigned mbids), so feel free to open a ticket and i look into adding that later.

<10volumetrique> lucifer: Found a ticket for it, not going to open a new one. https://tickets.metabrainz.org/browse/LB-1330?j... I guess other MBIDs are considered though?

LB-1330: Use Track/Release MBID in preference to auto-mapper if supplied in listen data

thanks 👍

<10volumetrique> at least, are recording_mbid and release_mbid used?

Ah yes, Gaga and typesense

<10volumetrique> I haven't dug through the source code...

Yes, we saw. But it's mostly a copypasta job, no?

yes should likely be.

volumetrique: mbid fields are not used while mapping. but if present recording_mbid/release_mbid data is preferred over what mapping assigns.

<10volumetrique> Okay, well I think I might deploy a lookup from track_mbid to recording+release_mbid in the meantime on my side

lucifer: yeah, I was thinking of implementing that endpoint

<10volumetrique> Thanks a lot

rimskii[m]: that sounds fine to me but you might have problems with testing it out.

i can create a small dump from apple music tables for you to test it locally i guess.

mayhem: thanks for the info. so the LB updates and search issues had all been related. Good to know

and thanks everyone involved for fixing :)

<lucifer> "rimskii: that sounds fine to..." <- ah i forgot about that

they were actually related, they are two different clusters of machines... if i understand correctly.

<lucifer> "i can create a small dump from..." <- that would be great!

is there any other way to implement that? to make it easier for testing i mean