I see the spotify_mapping table has a recording_msid field. Why MSID?
2022-01-12 01223, 2022
lucifer
with this we should be able to re run for last year
2022-01-12 01251, 2022
lucifer
my intent was that use msid for it. then go through msid->mbid mapping when we want mbid.
2022-01-12 01214, 2022
lucifer
because msid->mbid will improve continously. and some mappings may improve later on.
2022-01-12 01222, 2022
alastairp
yeah, great work over the break lucifer!
2022-01-12 01222, 2022
mayhem
I would need to make a corresponding YIM PR for the playlists for them to be repeatable, but no biggie.
2022-01-12 01258, 2022
mayhem
I need to ponder the spotify mapping a bit more.
2022-01-12 01217, 2022
lucifer
i have a use case for it. import loved tracks and playlist from spotify.
2022-01-12 01238, 2022
lucifer
makes sense
2022-01-12 01232, 2022
lucifer
oh! while we are here. i and alastairp had a brief chat on the LFM imports situation.
2022-01-12 01205, 2022
lucifer
so the YIM stuff revealed that a lot of people regularly submit to LFM but occasionally import to LB.
2022-01-12 01239, 2022
lucifer
this doesn't seem ideal. should we do something about this?
2022-01-12 01257, 2022
mayhem
> i have a use case for it. import loved tracks and playlist from spotify.
2022-01-12 01209, 2022
mayhem
this is for using MSID in the spotify mapping?
2022-01-12 01241, 2022
lucifer
no just having the mapping in general. basically being able to go from spotify track uri to msid/mbid
2022-01-12 01210, 2022
mayhem
I feel very strongly that we should never build anything new on MSID.
2022-01-12 01239, 2022
mayhem
it should always be MBIDs going forward. so unless we can find a use case that absolutely requires it, we should be using MBID
2022-01-12 01222, 2022
lucifer
i am not against that but we don't know how future mapping improvements will change msid to mbid. so i think it might be useful to keep msids around just for the sake of it till we have more clarity.
2022-01-12 01201, 2022
lucifer
plus say a bug causes msid mbid to be wrong and we later fix that, having the msid around lets propogate those fixes elsewhere.
2022-01-12 01242, 2022
lucifer
in the end we may end up not needing it anywhere but this feels like a low cost to bear especially for non user facing features.
2022-01-12 01209, 2022
mayhem
I'm not suggesting we get rid of MSIDs -- I'm just saying that we should always build new stuff for MBIDs, unless it is absolutely clear that we need to use MSIDs.
2022-01-12 01203, 2022
lucifer
right i agree.
2022-01-12 01237, 2022
mayhem
wrt to yim and lfm and imports. that pattern was pretty clear and is fine. doing anything about it would feel like coercion. the thing we should do is make LB much more attractive than LFM.
2022-01-12 01245, 2022
lucifer
makes sense.
2022-01-12 01222, 2022
lucifer
i was pondering if a continous importer like we have for spotify would be useful. setup it once and forget.
2022-01-12 01225, 2022
alastairp
lucifer had a suggestion to set up the lastfm importer like we have spotify - always running in the background (server-based, not browser-based) and continually updating people's listens. I don't think this is a good idea because it encourages people to keep treating lfm as the primary data source
2022-01-12 01230, 2022
alastairp
oh snap
2022-01-12 01235, 2022
lucifer
hah
2022-01-12 01213, 2022
mayhem ponders
2022-01-12 01210, 2022
mayhem
is this an opportunity to get rid of the current LFM importer, which is nothing but drama?
2022-01-12 01213, 2022
mayhem
if so, then YESS
2022-01-12 01228, 2022
lucifer
heh that's one incentive too.
2022-01-12 01233, 2022
alastairp
yes, right
2022-01-12 01248, 2022
mayhem
but, we need to get more intelligent about our importers.
2022-01-12 01253, 2022
lucifer
but keep in mind, we are adding another client based importer for the extended spotify importer.
2022-01-12 01205, 2022
mayhem
7 minutes to do an import where most users import nothing is not very intelligent.
2022-01-12 01222, 2022
mayhem
could we avoid that?
2022-01-12 01227, 2022
alastairp
technically I don't thing there are any negatives to the suggestion - we made the original importer browser-based in order to prevent possible issues with bulk API queries to lfm. but given their feedback when we were spamming requests with a single API key shows that this isn't a concern
2022-01-12 01250, 2022
mayhem
could we have two background importers running? one for long term one for real time?
2022-01-12 01252, 2022
lucifer
fwiw with LFM that isn't issue. LFM allows us to query stuff till the start of the user's history.
2022-01-12 01204, 2022
mayhem
exactly.
2022-01-12 01209, 2022
lucifer
with spotify if we stop, we may miss listens because we can only get last 50.
2022-01-12 01224, 2022
alastairp
yes, for spotify I think we can make the importer smarter to reduce time between updates
2022-01-12 01235, 2022
mayhem
agreed.
2022-01-12 01215, 2022
alastairp
I would probably treat them as 2 different importers - spotify_importer, lastfm_importer?
2022-01-12 01232, 2022
mayhem
with loads of shared code, I hope
2022-01-12 01249, 2022
lucifer
yes should be possible
2022-01-12 01227, 2022
lucifer
so LFM importer frontend goes away, we add one like spotify one for it.
2022-01-12 01236, 2022
mayhem
yeah
2022-01-12 01239, 2022
lucifer
👍
2022-01-12 01249, 2022
alastairp
yeah, only the "get listens from service" code would be differnt in that case
2022-01-12 01259, 2022
mayhem
ding
2022-01-12 01219, 2022
alastairp
hey, it could even be a single entrypoint with an arg saying which enum in the imports table to use and which "get lsitens" method
2022-01-12 01231, 2022
lucifer
yeah that but maybe add some delays to avoid running too frequently.
2022-01-12 01247, 2022
lucifer
separate container for each importer?
2022-01-12 01255, 2022
mayhem
yep, all that.
2022-01-12 01219, 2022
lucifer
sounds good.
2022-01-12 01226, 2022
mayhem
and I think we ought to try and predict when a user will need to be checked next.
2022-01-12 01237, 2022
mayhem
get a listen, set the next check for avg listen length.
2022-01-12 01204, 2022
mayhem
no listen, check less frequently.
2022-01-12 01237, 2022
lucifer
yeah makes sense
2022-01-12 01204, 2022
mayhem
and then check as theses timeouts are reached per user.
2022-01-12 01218, 2022
alastairp
lucifer: separate container, or 2 runit services in 1 container. implemetnation detail
yeah, users who import something their time around the loop should be priority for the next time around the loop. if they don't import (within time limits of the song), start falling down the priority queue. ensure that all users are checked every x time
2022-01-12 01222, 2022
lucifer
so this is about showing the artists when the use hovers on a country in the map.
2022-01-12 01239, 2022
mayhem
alastairp: that
2022-01-12 01245, 2022
lucifer
👍
2022-01-12 01213, 2022
mayhem
> Artist names from the same country are repeated because we explode the list of artist mbids when calculating countries. One possible solution is to only show the name only once and link it to the first artist mbid only. This is what we do elsewhere from artist links anyways.
2022-01-12 01235, 2022
mayhem
here I would be inclined to treat this as the artists individually and not the artist_credit.
2022-01-12 01253, 2022
lucifer
makes sense but we don't have the artist name available here.
2022-01-12 01203, 2022
mayhem
and this only list the artists that are actually associated with the given country.
2022-01-12 01215, 2022
lucifer
we only have the artist_credit_name.
2022-01-12 01238, 2022
mayhem
then we should switch the back-end to use an array of artists_mbids
2022-01-12 01250, 2022
mayhem
with an array of artist_names
2022-01-12 01256, 2022
mayhem
(or a dict of both)
2022-01-12 01215, 2022
lucifer
we do have artist mbids on the backend but getting artist names from artist mbids is an extra query.
2022-01-12 01223, 2022
lucifer
can we add this to the country endpoint?
2022-01-12 01242, 2022
lucifer
we currently don't have another use case for this anyways.
