what I've not seen in this discussion is an evaluation about what a good result is
2019-06-05 15609, 2019
ruaok
but I think exposing anything is premature. we will have a lot of iterations.
2019-06-05 15616, 2019
ruaok
but that can be in the cards for the future.
2019-06-05 15618, 2019
alastairp
regardless of what the process or end output is
2019-06-05 15648, 2019
ruaok
alastairp: a very good question. I've not thought about that yet.
2019-06-05 15604, 2019
ruaok
my focus so far has been to build the data sets that allow people to make recomendations.
2019-06-05 15631, 2019
ruaok
though, I think making a new DB and then allowing people to download dumps of it for our challenge in the fall makes a lot of sense.
2019-06-05 15657, 2019
ruaok
alastairp: does the industry have a metric for measuring the performance of rec systems?
2019-06-05 15607, 2019
alastairp
from my point of view, that's really important. since pristine__'s work has "something" working, but I've not seen any structured analysis as to whether the results are actually good
2019-06-05 15613, 2019
Nyanko-sensei joined the channel
2019-06-05 15620, 2019
alastairp
other than ruaok saying "well, that _does_ look like something that I'd want to listen to"
2019-06-05 15628, 2019
alastairp
right, there are 2 broad options
2019-06-05 15631, 2019
ruaok
correct, I agree.
2019-06-05 15609, 2019
alastairp
playlist recommendation (e.g. https://recsys-challenge.spotify.com/) evaluates you by withholding part of the playlist, and seeing how many of the items that you recommend are on the withheld part
2019-06-05 15616, 2019
ruaok
and really the CF stuff doesn't generate things I want to listen to. CF needs more backup/mashup.
2019-06-05 15627, 2019
alastairp
otherwise you have subjective analysis. give it to someone and ask them how good it is
2019-06-05 15602, 2019
alastairp
the first one is much easier to evaluate, but you end up basically only recommending people stuff that they already know
2019-06-05 15621, 2019
alastairp
because there's no other way of knowing that a recommendation out of their known songs is good for them
2019-06-05 15658, 2019
D4RK-PH0ENiX has quit
2019-06-05 15659, 2019
alastairp
so, alternatively, do similar to what gentlecat and philip did for their masters projects, generate a playlist, give it to someone, and ask them to thumbs up/down recommendations
2019-06-05 15625, 2019
ruaok
I really only see the latter as being possible. since we don't have 1M playlists to begin with.
2019-06-05 15628, 2019
alastairp
(then you have to work out how to fold that feedback into the algorithm too)
2019-06-05 15643, 2019
alastairp
you don't have playlists, but you have playback history
2019-06-05 15652, 2019
Nyanko-sensei has quit
2019-06-05 15628, 2019
D4RK-PH0ENiX joined the channel
2019-06-05 15650, 2019
ruaok
the CF alg will need to have a candidate dataset to recommend into.
2019-06-05 15612, 2019
ruaok
which we haven't quite sorted out to do create yet, but have some ideas.
2019-06-05 15653, 2019
ruaok
but that obviously impacts what gets generated. and may limit the effectiveness of using listens as a way of measuring effectiveness.
2019-06-05 15654, 2019
alastairp
so you want to build a test playlist? that's not a terrible idea
2019-06-05 15601, 2019
alastairp
but man, it's going to be so subjective
2019-06-05 15618, 2019
ruaok
it will be for sure.
2019-06-05 15633, 2019
ruaok
but I think that anything else is beyond the scope for the summer.
2019-06-05 15645, 2019
alastairp
ruaok: btw, bulk queries _do_ get slower, but it seems to be the transfer time for larger and larger responses rather than the actual db lookup
2019-06-05 15651, 2019
alastairp
so your nginx suggestion is good
2019-06-05 15600, 2019
alastairp
sure, not much time left in the summer for that
2019-06-05 15601, 2019
ruaok
if we get a page on LB where a user can click "gimme a playlist" and one appears in a reasonable amount of time, I would be happy for the summit.
2019-06-05 15610, 2019
ruaok
summer.
2019-06-05 15619, 2019
ruaok
alastairp: great.
2019-06-05 15653, 2019
ruaok
given how we're evolving all of this, this needs to be part of the roadmap for a challenge in the autumn.
2019-06-05 15604, 2019
ruaok
but for summer, it is too much.
2019-06-05 15611, 2019
ruaok
thanks for putting that on the radar, alastairp.
2019-06-05 15641, 2019
ruaok
iliekcomputers: pristine__: thoughts on this discussion?
2019-06-05 15632, 2019
iliekcomputers
not so much, evaluation is definitely something we need to work on soon.
2019-06-05 15604, 2019
iliekcomputers
i'd been thinking of how we could get user feedback (thumbs up/down) into the cf algorithm. i guess it'd involve adding/subtracting values into the listen counts passed into the cf algorithm.
2019-06-05 15636, 2019
ruaok
not sure if feeding back into CF is all that good to start with.
2019-06-05 15658, 2019
ruaok
feeding back into the rec alg itself might be better or easier to start with.
2019-06-05 15606, 2019
ruaok
or adjusting the candidate set.
2019-06-05 15633, 2019
iliekcomputers
hmm, yeah.
2019-06-05 15633, 2019
iliekcomputers
but no way of knowing that with no real evaluation so far. getting some recommendations into production with thumbs up / down should be priority for now, i guess.
2019-06-05 15657, 2019
ruaok
I also feel that if we get to the point where "I can't tell how much this decent recommendation is improving over time" then I'll be quite happy.
2019-06-05 15621, 2019
ruaok
which of course means that we need to have a more qualitative approach to evaluating recommendations.
2019-06-05 15650, 2019
reosarevok
You mean giving them to someone with better quality taste than ruaok? Ok, me and zas are available :p
2019-06-05 15658, 2019
iliekcomputers
to be honest, we can't tell that right now either, really.
2019-06-05 15615, 2019
ruaok
both of you are right.
2019-06-05 15627, 2019
ruaok
but I haven't seen anything that made me smile yet.
2019-06-05 15645, 2019
ruaok
only things that I am convinced that I don't want to listen to.
2019-06-05 15648, 2019
reosarevok pats ruaok on the head
2019-06-05 15617, 2019
ruaok
of course, we're also still early in the game.
2019-06-05 15634, 2019
reosarevok
Very much so
2019-06-05 15607, 2019
reosarevok
Qualitative evaluation is going to be very hard anyway, because it depends on having a lot of people with different tastes say "this, this is good shit"
2019-06-05 15616, 2019
ruaok
I guess if we can't please ourselves on a very basic level, then a more quantitative solution will only confirm what we already know.
2019-06-05 15617, 2019
reosarevok
We barely have a lot of people *submitting* yet :)
2019-06-05 15619, 2019
ruaok
(read: we suck)
2019-06-05 15634, 2019
ruaok
yeah, that is another issue that I am grappling with.
2019-06-05 15636, 2019
reosarevok
Wait
2019-06-05 15637, 2019
reosarevok
"which of course means that we need to have a more qualitative approach to evaluating recommendations."
2019-06-05 15640, 2019
reosarevok
Did you mean quantitative?
2019-06-05 15649, 2019
ruaok
we keep releasing stuff and focusing on the next thing, but we need to work to get more users.
2019-06-05 15609, 2019
ruaok
qualitative, I guess.
2019-06-05 15624, 2019
reosarevok
Oh, ok
2019-06-05 15631, 2019
ruaok
My brain is barely cohesive this morning. feh. jetlag gets worse as one ages.
2019-06-05 15600, 2019
reosarevok
If you can come up with some half-decent quantitative / programmatical way of knowing if stuff is kinda-sorta improving, that would be great, if only because for a human is hard to tell I feel
2019-06-05 15609, 2019
reosarevok
"Ok, I still hate this shit, but do I hate it LESS?"
2019-06-05 15618, 2019
ruaok
no arguments from me.
2019-06-05 15653, 2019
ruaok
still, I'm happy we're facing these issues/questions.
2019-06-05 15657, 2019
ruaok
clearly a sign of progress.
2019-06-05 15650, 2019
reosarevok
"Just how shit are we really?" "PROGRESS!"
2019-06-05 15651, 2019
reosarevok
:D
2019-06-05 15651, 2019
iliekcomputers
did we come to a conclusion about storing the data?
2019-06-05 15653, 2019
reosarevok
But yeah, I guess :)
2019-06-05 15602, 2019
ruaok
iliekcomputers: no
2019-06-05 15609, 2019
iliekcomputers
😂
2019-06-05 15623, 2019
reosarevok
iliekcomputers: you're a playground bully :p
2019-06-05 15631, 2019
reosarevok
You guys have more money than everyone else combined!
2019-06-05 15645, 2019
ferbncode
iliekcomputers: 😂
2019-06-05 15610, 2019
reosarevok
You'll still manage to lose to Pakistan somehow anyway, though, so it's ok
2019-06-05 15639, 2019
alastairp
iliekcomputers: I want to add a constant from somewhere in the code into a sphinx documentation so that it shows up in the api documentation. ever done that?
anyway. ruaok I wanted to ask you a slight off topic question. how hard is it really to register and own and maintain a *.cat domain (seeingas you live in barceloan now i thoguht you woudl know, don't you also have a *.cat websie now?)
2019-06-05 15614, 2019
ruaok
easy in the grand scheme of things.
2019-06-05 15628, 2019
ruaok
there is one caveat -- there needs to be some catalan content on the page.
2019-06-05 15632, 2019
CatQuest
exactly
2019-06-05 15639, 2019
CatQuest
but liek, how strict are they?
2019-06-05 15655, 2019
ruaok
my mayhem.cat page has no text, except for "Benvinguts". so, welcome in Catalan. No one has ever come complaining.
2019-06-05 15601, 2019
CatQuest
if I translated reosarevok's "nokkloom" page into estonian, wil lit suffice
