_lucifer: hmm, I don't recall having that conversation. But good point. not sure why we are using the list version in all of these methods that just take a single key as argument
alastairp: no i haven't. because the content will depend on how the common script to use. so i am waiting for the common script before i add the .finish files.
alastairp
right. no problem. I'm on it
_lucifer
ruaok: recs, similar users and stats all generated. a number of stats job will run again tonight so we'll know by tomorrow if there is some other issue with the cluster. otherwise its all ready.
counters aren't encoded, because we rely on redis to actually do the modification
feh, so you'll have to do `cache.get(whatever, decode=False)`
_lucifer: I still don't like that, even though we decided to go with it
_lucifer
alastairp, we decided to add encoding but default to false, right?
alastairp
I'll have to look at the chat logs again. I thought we were going to automatically encode/decode text values, and have the choice up to the user for the bulk get options
ruaok
should `cache.set(REDIS_USER_LISTEN_COUNT + user_name, 0, time=0)`have encode=False then too?
alastairp
I think I'll have to have a chat with ruaok about this on wednesday
ruaok: ah. so you're setting a value manually, then you want to incr, then you want to get?
ruaok
the boom in the paste above.
alastairp
yes, you'll need encode=False
so all of today's discussion makes it clear that this is super fragile and not at all intuitive
ruaok nods a lot
the context is: the cache is supposed to be magic and you're supposed to be able to go "cache x object" and it'll encode it to bytes, and then decode it seamlessly when you want the value back out
which works OK when you're just treating it as a kv store where you want to put strings, or integers
ruaok
tasty magic
alastairp
but now we're trying to actually use redis as redis
ruaok
but gets muddled with inc
alastairp
yeah, so things like inc, or hashes, or sets introduce additional complexity
oh, I just had an epiphany
we had this discussion about cache/data store split
what if we had the cache module, which is _just_ get/set, and does serialisation. and the data module which basically exposes redis, and doesn't do any serialisation
it's clear that ruaok's usage is data storage. he doesn't need any magic here (other than what redis provides)
ruaok
yes, something like that.
alastairp
so turns out that iliekcomputers was correct 3 years ago
!m iliekcomputers
BrainzBot
You're doing good work, iliekcomputers!
ruaok
because there are other functions about redis we want to use and we'll constantly have to keep BU up to date.
alastairp
yeah, every time we need another datatype we have to add more methods which just call exactly the same redis method
iliekcomputers
\o/
Correct about what?
alastairp
does it matter? ;)
ruaok: maybe we can grab some time on Weds to ensure that we go about this correctly
iliekcomputers
Lol
ruaok
k
alastairp
iliekcomputers: some time ago you suggested a "data" module in BU for storing long-term data in redis
People up for reviews: alastairp, ruaok, reosarevok, Freso, zas, yvanzo, Mr_Monkey, bitmap, _lucifer, CatQuest, shivam-kapila – anyone else who want to give review, let me know ASAP. :)
I haven’t received any mailed‐in reviews, so…
alastairp: Go!
alastairp
hello
This week I worked with _lucifer to finish off the last breaking api changes in BU in order to release BU 2.0. This is mostly finished, pending a small discussion about redis
last Wednesday we found a bug in the listen processing pipeline that was causing listens to get stuck, we made a quick fix for the bug, but it openend up some more validation tasks that we have to do during listen reading
_lucifer
(I am a bit busy rn. can i please go last?)
alastairp
with Mr_Monkey we also looked into some other issues surrounding incoming listen validation (which fields are required and what values they can have) and how we can improve that in some cases
Mr_Monkey
The Strange Case of the Missing Track Title
alastairp
finally, I ran some tests to get AB dumps and incremental dumps working, that's ongoing
Mr_Monkey: thanks for speaking up, you can go next
Mr_Monkey
Hello !
Last week I worked on ListenBrainz listens pagination along with ruaok, and also had a look at why the Spotify player is currently not working on LB. Sigh, neither of the main players are working at the moment…
Turns out the Spotify API is still in beta and they don't seem to announce changes
So I'm expecting more of that until they hit a stable release
ruaok
oh, that is convenient!
Mr_Monkey
Also reviewed GSoC application drafts adn lots of PRs !
Also fixed LB-864 real quick, that was an easy one for once