this whole neil diamond thin is showing spotify's stripes. that and our GDPR push just shows me loads of things I do not like.
2022-01-29 02914, 2022
mayhem
lucifer: I think I have another hour or so of work to do on the recording_similarity work. after that, all the work on the python code is pretty pointless.
2022-01-29 02905, 2022
mayhem
but, there is a proof of concept which shows that we should investigate this more, since it could also produce artist similarities and possibly even release similarity.
2022-01-29 02939, 2022
mayhem
if you feel like brainstorming the spark move at some point, let me know.
2022-01-29 02903, 2022
mayhem
windowing functions sound like a very good start for computing this data.
2022-01-29 02936, 2022
Sophist-UK joined the channel
2022-01-29 02944, 2022
Sophist-UK has quit
2022-01-29 02930, 2022
mayhem
I guess my understanding on how spark distributes in memory tasks is unclear to me. I suppose we need to break the task into smaller chunks, perhaps 5 years worth of data and dump it into temporary dataframes.
2022-01-29 02900, 2022
mayhem
then at the end we need to collate all the results into one. that step, I have no clue about.
2022-01-29 02914, 2022
aerozol has quit
2022-01-29 02904, 2022
trolley has quit
2022-01-29 02928, 2022
trolley joined the channel
2022-01-29 02907, 2022
v6lur joined the channel
2022-01-29 02943, 2022
lucifer
mayhem: hi! yeah i am not much familiar with how to get this all to work in spark, my first try is going to be on these lines:
for each lookahead, repeat the query append the grouped by results and do a final group by and deduplicate.
2022-01-29 02941, 2022
lucifer
so N passes over the listen table, another way could be to make columns like row_2 and similar_2 have the multiple windows in the same query.
2022-01-29 02941, 2022
lucifer
(also does the query make sense?)
2022-01-29 02956, 2022
MRiddickW joined the channel
2022-01-29 02905, 2022
v6lur has quit
2022-01-29 02904, 2022
v6lur joined the channel
2022-01-29 02922, 2022
Sophist-UK joined the channel
2022-01-29 02912, 2022
Sophist-UK has quit
2022-01-29 02918, 2022
Ansh
Hi lucifer I am quite interested in the critiquebrainz project. Can you pls guide me how I can contribute to it.
2022-01-29 02918, 2022
lucifer
hi Ansh. You are already on the right track! thanks for the recent critiquebrainz PRs. if you are looking for more tickets to work on, you can take up any bugfix ticket from the issue tracker and work on it. there are also many enhancement/feature tickets, if you'd like to work on one of those we can discuss about those first as we may not want to implement some of those.
2022-01-29 02928, 2022
Ansh
Sure! I'll look for more tickets and will discuss about them here first. Thanks!
2022-01-29 02943, 2022
lucifer
great!
2022-01-29 02910, 2022
Shubh has quit
2022-01-29 02956, 2022
texke`_ has quit
2022-01-29 02920, 2022
akshaaatt
lucifer the listens submission is now working fine from the app! Did you fix something on the backend? Thanks btw
