do you by any chance have your previous year's spotify report?
2020-09-22 26658, 2020
iliekcomputers
i could find examples, sure.
2020-09-22 26606, 2020
ishaanshah
I joined spotify in March, I would like to get a rough idea
2020-09-22 26627, 2020
ruaok
maybe make them not quite so garish. :D
2020-09-22 26617, 2020
iliekcomputers
it's not very insightful tbh
2020-09-22 26635, 2020
iliekcomputers
mostly because spotify doesn't really expose top artists etc as such
2020-09-22 26643, 2020
iliekcomputers
so seeing them yearly is something users find useful
2020-09-22 26651, 2020
iliekcomputers
i'm pretty sure we can do much better
2020-09-22 26625, 2020
ruaok nods
2020-09-22 26634, 2020
iliekcomputers
looks like they've gotten rid of it now
2020-09-22 26645, 2020
ishaanshah
ohk I saw some examples of it on the net
2020-09-22 26647, 2020
iliekcomputers
it was called spotify wrapped
2020-09-22 26651, 2020
ruaok
even including the 2020 heatmap and globe would be better already
2020-09-22 26655, 2020
ishaanshah
its mostly the stuff we already do
2020-09-22 26646, 2020
iliekcomputers
pretty much, yeah.
2020-09-22 26653, 2020
ishaanshah
So what I was thinking of is, we add some more statistics to the current ones
2020-09-22 26608, 2020
ishaanshah
Top discoveries is interesting
2020-09-22 26610, 2020
iliekcomputers
i think what we could do is this
2020-09-22 26619, 2020
shivam-kapila
I think I might have an SS
2020-09-22 26627, 2020
iliekcomputers
change the reports page to show last year's report as well
2020-09-22 26609, 2020
iliekcomputers
this would take a bunch of work i assume, but if we do this, we should make it so that in the future we can essentially show any year's, month's or week's report if we want to, easily
2020-09-22 26622, 2020
iliekcomputers
and then just add more cool stuff to the reports page
2020-09-22 26642, 2020
iliekcomputers
and then near the end of december, send people a link to their 2019 report page
2020-09-22 26653, 2020
shivam-kapila
2020*
2020-09-22 26655, 2020
ishaanshah
> this would take a bunch of work i assume, but if we do this, we should make it so that in the future we can essentially show any year's, month's or week's report if we want to, easily
2020-09-22 26656, 2020
ishaanshah
A lot of this would be easier if we move to db to relational
2020-09-22 26600, 2020
iliekcomputers
ishaanshah: yep
2020-09-22 26617, 2020
iliekcomputers
2020, yes
2020-09-22 26649, 2020
ishaanshah
yep that makes sense
2020-09-22 26607, 2020
ishaanshah
that way we can be sure that we get some result by year end
2020-09-22 26622, 2020
ishaanshah
and if time permits we could do some extra cool stuff too
2020-09-22 26625, 2020
iliekcomputers
yep.
2020-09-22 26641, 2020
ishaanshah
like 5x5 discovery, streak...
2020-09-22 26623, 2020
ishaanshah
this sounds like a good plan
2020-09-22 26611, 2020
iliekcomputers
awesome, i still think it'd make sense to write a design doc before we actually start implementing
2020-09-22 26613, 2020
ishaanshah
so our first goal should be moving top entities to relational db?
2020-09-22 26637, 2020
ishaanshah
> awesome, i still think it'd make sense to write a design doc before we actually start implementing
2020-09-22 26637, 2020
ishaanshah
yep yep, agreed especially with the DB stuff
2020-09-22 26608, 2020
shivam-kapila
i got it
2020-09-22 26611, 2020
iliekcomputers
>our first goal should be moving top entities to relational db?
2020-09-22 26622, 2020
iliekcomputers
i'll leave it up to you, there are multiple ways we can do this, maybe we start by adding the new stats in a relational db and then later we migrate the existing stats. either would work i guess
2020-09-22 26655, 2020
ishaanshah
new stats?
2020-09-22 26613, 2020
iliekcomputers
like if you wanna do the discovery, streak things too
2020-09-22 26622, 2020
ishaanshah
ah ohk
2020-09-22 26652, 2020
ishaanshah
I think that would be easier if we move top entities to relational db
2020-09-22 26659, 2020
ishaanshah
shivam-kapila: thanks :)
2020-09-22 26605, 2020
iliekcomputers
hmm, ok.
2020-09-22 26620, 2020
iliekcomputers
let's try to come up with a design that allows for future use cases too
2020-09-22 26633, 2020
iliekcomputers
for example, maybe we want to show people a report from 5 years ago
2020-09-22 26645, 2020
iliekcomputers
or maybe users want to see their report for a particular month
2020-09-22 26647, 2020
iliekcomputers
things like that
2020-09-22 26603, 2020
ishaanshah
yes, I had thought about this before
2020-09-22 26622, 2020
ishaanshah
I am not sure how fast postgres is at aggregating data
2020-09-22 26606, 2020
ishaanshah
but if we do top artist per day for each user, we can get stats for any time range
2020-09-22 26640, 2020
iliekcomputers
hmm, i think that would be a bit too much calculation
2020-09-22 26611, 2020
iliekcomputers
i think we'll want to keep the calculation as is on the spark side, and just store things differently
2020-09-22 26624, 2020
iliekcomputers
for example, right now we store it in json in a "year" key
2020-09-22 26605, 2020
iliekcomputers
instead what we should probably do is have a time_range column which is an enum (day, month, year, all_time) and a time_range_value column which could be set to 2018, 2019 etc or something
2020-09-22 26642, 2020
sumedh joined the channel
2020-09-22 26623, 2020
ishaanshah
yep that makes sense
2020-09-22 26651, 2020
ishaanshah
if we index our table properly
2020-09-22 26608, 2020
ishaanshah
we could use the same table to get data for entity pages in future
2020-09-22 26616, 2020
ishaanshah
and for sitewide data
2020-09-22 26628, 2020
ishaanshah
cool, this sounds good, I will spend some time thinking about the technical details then
2020-09-22 26633, 2020
iliekcomputers
awesome, thanks.
2020-09-22 26640, 2020
ishaanshah
exact schama and stuff
2020-09-22 26654, 2020
iliekcomputers
mhmm. sounds good.
2020-09-22 26614, 2020
ishaanshah
I think we should be able to do the migration + 1 or 2 extra stats
2020-09-22 26618, 2020
ishaanshah
cool, one last test for the spark import PR is remaining
2020-09-22 26629, 2020
ishaanshah
I'll try to get it done tomorrow
2020-09-22 26632, 2020
iliekcomputers
awesome, excited to merge that.
2020-09-22 26641, 2020
ishaanshah
yep
2020-09-22 26654, 2020
ishaanshah
did you get a chance to check the full dump issue?
2020-09-22 26624, 2020
iliekcomputers
not yet. but the graph you linked made it pretty obvious that it was a space issue
