this whole neil diamond thin is showing spotify's stripes. that and our GDPR push just shows me loads of things I do not like.
lucifer: I think I have another hour or so of work to do on the recording_similarity work. after that, all the work on the python code is pretty pointless.
but, there is a proof of concept which shows that we should investigate this more, since it could also produce artist similarities and possibly even release similarity.
if you feel like brainstorming the spark move at some point, let me know.
windowing functions sound like a very good start for computing this data.
Sophist-UK joined the channel
Sophist-UK has quit
I guess my understanding on how spark distributes in memory tasks is unclear to me. I suppose we need to break the task into smaller chunks, perhaps 5 years worth of data and dump it into temporary dataframes.
then at the end we need to collate all the results into one. that step, I have no clue about.
aerozol has quit
trolley has quit
trolley joined the channel
v6lur joined the channel
lucifer
mayhem: hi! yeah i am not much familiar with how to get this all to work in spark, my first try is going to be on these lines:
for each lookahead, repeat the query append the grouped by results and do a final group by and deduplicate.
so N passes over the listen table, another way could be to make columns like row_2 and similar_2 have the multiple windows in the same query.
(also does the query make sense?)
MRiddickW joined the channel
v6lur has quit
v6lur joined the channel
Sophist-UK joined the channel
Sophist-UK has quit
Ansh
Hi lucifer I am quite interested in the critiquebrainz project. Can you pls guide me how I can contribute to it.
lucifer
hi Ansh. You are already on the right track! thanks for the recent critiquebrainz PRs. if you are looking for more tickets to work on, you can take up any bugfix ticket from the issue tracker and work on it. there are also many enhancement/feature tickets, if you'd like to work on one of those we can discuss about those first as we may not want to implement some of those.
Ansh
Sure! I'll look for more tickets and will discuss about them here first. Thanks!
lucifer
great!
Shubh has quit
texke`_ has quit
akshaaatt
lucifer the listens submission is now working fine from the app! Did you fix something on the backend? Thanks btw