well, ok. we HAD a check. there is a define for that.
2021-10-26 29931, 2021
ruaok
lucifer: if you put in the check, I'll clean up the DB.
2021-10-26 29940, 2021
lucifer
on it
2021-10-26 29929, 2021
lucifer
ruaok: for the error to show, how does this sound? "Value for key listened_at is too low. listened_at value should be of 2002 (Last.fm founding year) or later"
2021-10-26 29943, 2021
ruaok
hmmm.
2021-10-26 29936, 2021
ruaok
LAST_FM_FOUNDING_YEAR = 2002
2021-10-26 29953, 2021
ruaok
hmmm. I have a recollection that that ought to be 2005
alastairp: I have a vague memory of RJ telling us to ignore everything prior to a certain date.
2021-10-26 29922, 2021
ruaok
founding year was defined in the main codebase at some point. now its just in spark, which seems odd since listen validation is being done on the flask side.
2021-10-26 29935, 2021
lucifer
yeah
2021-10-26 29902, 2021
lucifer
i think at some point the check got deleted.
2021-10-26 29928, 2021
lucifer
then stuff got moved around as usual refactoring.
2021-10-26 29900, 2021
ruaok
I want find the place where we had defined it orginally
2021-10-26 29914, 2021
ruaok
there was an epoch timestamp as well.
2021-10-26 29957, 2021
lucifer
trying to find that in git history.
2021-10-26 29915, 2021
ruaok
us both
2021-10-26 29958, 2021
monkey is going to deploy huesound to test.LB, making some final UI refinements
also +100 n the buttons for saying "this is not correct" and a button for saying "this is the one it really is"
2021-10-26 29945, 2021
CatQuest
<333
2021-10-26 29921, 2021
ruaok thinks about scalabilty for the incremental dumps
2021-10-26 29943, 2021
CatQuest
also re listen/scrobble dates. there are tools for manually adding listened stuff to (last.fm) where you could conceivably add something listened to in 1998 to l.fm
2021-10-26 29907, 2021
ruaok
lucifer: one thing I hadn't yet considered with the "save incremental listens to a file" approach was that it makes scaling timescale_writer much much harder.
2021-10-26 29927, 2021
ruaok
so, I am going back to looking at being able to sort on created, rather than listened_at.
I'm going to try creating an index on created, but I think that in order for timescale to use it, we'd need to upgrade ts.
2021-10-26 29917, 2021
lucifer
yeah, i had that in mind ts_writer could be impacted. but i was unsure about the exact numbers.
2021-10-26 29900, 2021
ruaok
if the spark cluster could do it at the end of the unique stream, that would be better.
2021-10-26 29903, 2021
lucifer
we have to upgrade ts anyways sooner or later so that's fine
2021-10-26 29922, 2021
ruaok
but, let me investigate this. I hope this will help and let us skip this extra complexity.
2021-10-26 29924, 2021
lucifer
can you elaborate on that?
2021-10-26 29945, 2021
ruaok
on this ? "if the spark cluster could do it at the end of the unique stream, that would be better."
2021-10-26 29948, 2021
lucifer
yes
2021-10-26 29910, 2021
lucifer
oh nvm, i get it now. connect to rmq as a consumer?
2021-10-26 29923, 2021
ruaok
because the unique stream could come from multiple timescale_writers.
2021-10-26 29929, 2021
ruaok
yes, that.
2021-10-26 29947, 2021
ruaok
and then the files are already there.
2021-10-26 29904, 2021
lucifer
sure that's a good plan in the long term. not sure we need that now.
2021-10-26 29931, 2021
ruaok
we dont need multuple timescale writers yet, no.
2021-10-26 29952, 2021
lucifer
i think adding a container to do this on kiss/gaga and shipping to spark is a good middle ground for now.
2021-10-26 29909, 2021
ruaok
but we must resist fucking future ruaok and lucifer. because those two will need to wake up in the middle of the night and deploy more timescale-writers.
2021-10-26 29936, 2021
ruaok
"i think adding a container to do this on kiss/gaga and shipping to spark is a good middle ground for now."
2021-10-26 29959, 2021
ruaok
it is tempting to think that way. but we're pushing an impending disaster ahead of us. knowingly.
2021-10-26 29904, 2021
lucifer
oh definitely need to avoid that.
2021-10-26 29911, 2021
ruaok
yeah.
2021-10-26 29921, 2021
lucifer
yes but we don't have the MB on spark yet and just getting that to spark is huge work
2021-10-26 29929, 2021
ruaok
so, if we NEED to make this work differently RMQ on J5 is the way to go.
2021-10-26 29940, 2021
ruaok
yeah, loads.
2021-10-26 29945, 2021
lucifer
but if that needs to be done then let's try to plan it then once alastairp is around or later this week?
2021-10-26 29942, 2021
lucifer
even if don't implement rn, having an idea of how to get about it would be good.
[listenbrainz-server] 14mayhem opened pull request #1678 (03master…better-incremental-dumps): (LB-980) AISOTT: Improve incremental dumps by sorting on created, rather than listened_at https://github.com/metabrainz/listenbrainz-server…
2021-10-26 29904, 2021
Lotheric has quit
2021-10-26 29928, 2021
Lotheric joined the channel
2021-10-26 29929, 2021
ruaok
overheard in catalunya: "I made fideuà like a biryani and now I don't know whom to hide from"
ah ok. i asked because i remember having this discussion with you about another delete endpoint (unfollow user iirc) where we decided its fine to return 200 if nothing was deleted.
2021-10-26 29936, 2021
alastairp
oh yes, that's right
2021-10-26 29924, 2021
lucifer
this is a different endpoint but we should probably be consistent unless there's a reason not to be.
2021-10-26 29941, 2021
alastairp
so the question is: when is it useful for a user to know the difference between something being successfully deleted, and something not existing?
2021-10-26 29945, 2021
lucifer
yup right. i cannot think of a use case currently where the distinction would matter.
2021-10-26 29903, 2021
aerozol has quit
2021-10-26 29925, 2021
aerozol joined the channel
2021-10-26 29900, 2021
lucifer
but get_or_create can be a close analogy. we have that in our own codebase and also use cases where the distinction is useful.
2021-10-26 29952, 2021
opal has quit
2021-10-26 29930, 2021
opal joined the channel
2021-10-26 29942, 2021
lucifer
i suggest let's document the current behvaiour and consider again if someone complains.
2021-10-26 29921, 2021
monkey
akshat: Yes, I noticed the huesound page wasn't working on mobile. I did work on it this morning and improved it some. Just deployed the result if you want to have a look
2021-10-26 29930, 2021
ruaok
alastairp: any chance you might have some time for the huesound PR today?
2021-10-26 29934, 2021
ruaok
it would be nice to get that out soon
2021-10-26 29914, 2021
alastairp
yes, should be no problem. let me just finish some other reviews of lucifer's first
2021-10-26 29923, 2021
ruaok
👍
2021-10-26 29933, 2021
lucifer
:-D
2021-10-26 29940, 2021
alastairp
ruaok: did I see that you were adding an index to listens table?
