yvanzo: can you check sir queues? We still have many entries in search failed, and it keeps growing. Not sure how to interpret this.
pranav[m]
aerozol: re playlists, I would first like to complete the core features of the user pages first and then do the playlists tab
zerodogg has quit
pranav[m] uploaded an image: (1141KiB) < https://matrix.chatbrainz.org/_matrix/media/v3/download/matrix.org/bzzKuSEfpBLKXrytiJYvdvJe/Screenshot%202024-06-17%20at%203.01.29%E2%80%AFPM.png >
aerozol: Could u lemme know what is to be done when user is already followed? Just need color and text of that button
rimskii[m]
reosarevok: hi! I wanted to ask if you can clarify some things regarding spotify lookups ( I remember you mentioned that we can ask you for help regarding MusicBrainz if needed.)
I’ve checked the spotify_lookup_from_mbid api and it seems it works by looking for an exact track match using the MBID to fetch the Spotify ID. Is that correct? Does that mean that each recording in MB contains spotify id?
I was thinking how can I implement the same thing for apple music for example
zerodogg joined the channel
d4rk joined the channel
d4rkie has quit
reosarevok[m] joined the channel
reosarevok[m]
Hi! That wasn't written by the MusicBrainz team I think so I'd have to check, can you point me to the code?
mayhem[m] joined the channel
mayhem[m]
lucifer: please let me know when you're about, I'd love a quick catchup regarding spotify playlists and stats issues.
If you're not sure, no worries at all! I guess it's outside of MB
yvanzo[m]
Hi zas, on it.
mayhem[m]
yvanzo: zas can one of you catch me up to the current status of the solr cluster? is there anything I can do to help it become more stable?
yvanzo[m]
mayhem, zas : SIR is currently indexing documents at full speed again.
mayhem[m]
have we switched to the new cluster yet?
reosarevok[m]
rimskii: I think mayhem is the person for this :)
mayhem[m]
sorry for not keeping up, but last week sleep was utter hell. I feel somewhat normal today.
rimskii[m]
reosarevok[m]: okay thanks ! :)
Sorry for bothering you
yvanzo[m]
mayhem: Not yet, we want to do it these days if not today. But we might have to keep the old cluster in parallel for a month or so.
mayhem[m]
ok, that's good. so all the problems, in theory, will go away soon? 🤞
rimskii: let me see if I can help you. we do not have 1:1 mapping of MBIDs to Spotify IDs.
(we wish, but alas no)
rimskii[m]
<rimskii[m]> "reosarevok: hi! I wanted to..." <- > <@kubrimskii:matrix.org> reosarevok: hi! I wanted to ask if you can clarify some things regarding spotify lookups ( I remember you mentioned that we can ask you for help regarding MusicBrainz if needed.)... (full message at <https://matrix.chatbrainz.org/_matrix/media/v3/...>)
mayhem[m]
yes, sure.
rimskii[m]
i say you already answered haha
s/say/saw/
mayhem[m]
the goal is to match a MBID to an apple music id, yes?
lets step through this and see if we can work it out.
the first two "with" statements do:
1. Resolve redirects: MBIDs can at times be deprecated and point to newer MBIDs. This step ensures that the given MBIDs are currently valid MBIDs (e.g. they are valid or were redirected)
zerodogg has quit
2. Resolve recording MBIDs to "canonical recording" MBIDs. A recording of a song may have more than one actual recording in MB, so this step ensures that we find the recording we consider to be "canonical". Canonical is important because we can find the metadata for each canonical recording easily in the metadata cache.
so in the for loop, we check the redirects, then the canonicals, then fetch the metadata.
all this was basically to take an MBID and resolve it to the correct metadata for that MBID.
because that is how the spotify data index is created.
then, finally the look up is done via the lookup_using_metadata(ordered_metadata) function.
<mayhem[m]> "the first two "with" statements..." <- well If im going to do such lookup for apple music should I do the same here too?
mayhem[m]
so in the utils module I just linked, it attempts to find the right spotify matches. and note that it does not do that with spotify ids, but with metadata lookups.
I think in the end you'll pretty much copy everything that was done for spotify to the apple setup.
rimskii[m]
haha ok
ok
if I remember correcly lucifer said we do have for apple music
so far it seems that all the code between apple and spotify is wholly separate, so copying the spotify code and modifying it for apple seems to be the right approach.
rimskii[m]
i see
So as far as I got, the spotify_lookup first finds the right mbid then searches for spotify metadata, right?
mayhem[m]
yes
rimskii[m]
mayhem[m]: spotify metadata is spotify_cache here, right?
mayhem[m]
yes
rimskii[m]
ahh i see
I got it
thank you soo much, appreciate it much !! :)
mayhem[m]
no problem.
lucifer
mayhem[m]: hi!
rimskii[m]: will check and let you know in a bit.
mayhem[m], outsidecontext: spark cluster was unavailable last week for a few days because of memory and storage issues apparently. i reset-ed the cluster on the weekend and did a fresh install, its processing requests as we speak. stats are done, recommendations and playlists queued.
mayhem[m]: the deleteme hetzner can be removed now.
i have setup the ansible playbooks for the spark cluster and used it to do the fresh installs.
mayhem[m]
Lucifer: one request -- when you expect downtime for the cluster, ping me please. So I can tweet/post about doing maintenance.
lucifer
mayhem[m]: it was unexpected.
i discovered it when diagnosing why stats were not updating.
after discovering it, i got to fixing it but that ended up taking over a day.
mayhem[m]
Ah, another case for more monitoring. But next time you do notice, please let me know
lucifer
yes sure
mayhem[m]
Deleteme server, deleted. Thanks.
lucifer
(to be clear, i discovered the memory and storage issues first and then reset it, not the other way around)
btw we should be able to add new servers to the cluster with ease now.
should be handy for trying out MLHD data processing stuff.
mayhem[m]
Oh yisss! Great news!
shall we do that after south or even before?
s/south/oauth/
lucifer
i guess after might be better
have to also deal with LB gaga disks and typesense
zerodogg joined the channel
Jigen, bitmap[m]: restored the data. thanks!
volumetrique: yes, we currently don't consider track_mbids during mapping.
but iirc mayhem[m] and i discussed previously to change that (not to consider track mbids in mapping but prefer track mbids if submitted over mapping assigned mbids), so feel free to open a ticket and i look into adding that later.