_lucifer: hmm, I don't recall having that conversation. But good point. not sure why we are using the list version in all of these methods that just take a single key as argument
alastairp: no i haven't. because the content will depend on how the common script to use. so i am waiting for the common script before i add the .finish files.
2021-04-19 10914, 2021
alastairp
right. no problem. I'm on it
2021-04-19 10934, 2021
_lucifer
ruaok: recs, similar users and stats all generated. a number of stats job will run again tonight so we'll know by tomorrow if there is some other issue with the cluster. otherwise its all ready.
counters aren't encoded, because we rely on redis to actually do the modification
2021-04-19 10907, 2021
alastairp
feh, so you'll have to do `cache.get(whatever, decode=False)`
2021-04-19 10921, 2021
alastairp
_lucifer: I still don't like that, even though we decided to go with it
2021-04-19 10947, 2021
_lucifer
alastairp, we decided to add encoding but default to false, right?
2021-04-19 10934, 2021
alastairp
I'll have to look at the chat logs again. I thought we were going to automatically encode/decode text values, and have the choice up to the user for the bulk get options
2021-04-19 10949, 2021
ruaok
should `cache.set(REDIS_USER_LISTEN_COUNT + user_name, 0, time=0)`have encode=False then too?
2021-04-19 10949, 2021
alastairp
I think I'll have to have a chat with ruaok about this on wednesday
2021-04-19 10911, 2021
alastairp
ruaok: ah. so you're setting a value manually, then you want to incr, then you want to get?
2021-04-19 10927, 2021
ruaok
the boom in the paste above.
2021-04-19 10935, 2021
alastairp
yes, you'll need encode=False
2021-04-19 10955, 2021
alastairp
so all of today's discussion makes it clear that this is super fragile and not at all intuitive
2021-04-19 10949, 2021
ruaok nods a lot
2021-04-19 10933, 2021
alastairp
the context is: the cache is supposed to be magic and you're supposed to be able to go "cache x object" and it'll encode it to bytes, and then decode it seamlessly when you want the value back out
2021-04-19 10950, 2021
alastairp
which works OK when you're just treating it as a kv store where you want to put strings, or integers
2021-04-19 10954, 2021
ruaok
tasty magic
2021-04-19 10906, 2021
alastairp
but now we're trying to actually use redis as redis
2021-04-19 10907, 2021
ruaok
but gets muddled with inc
2021-04-19 10922, 2021
alastairp
yeah, so things like inc, or hashes, or sets introduce additional complexity
2021-04-19 10933, 2021
alastairp
oh, I just had an epiphany
2021-04-19 10942, 2021
alastairp
we had this discussion about cache/data store split
2021-04-19 10939, 2021
alastairp
what if we had the cache module, which is _just_ get/set, and does serialisation. and the data module which basically exposes redis, and doesn't do any serialisation
2021-04-19 10913, 2021
alastairp
it's clear that ruaok's usage is data storage. he doesn't need any magic here (other than what redis provides)
2021-04-19 10935, 2021
ruaok
yes, something like that.
2021-04-19 10951, 2021
alastairp
so turns out that iliekcomputers was correct 3 years ago
2021-04-19 10953, 2021
alastairp
!m iliekcomputers
2021-04-19 10953, 2021
BrainzBot
You're doing good work, iliekcomputers!
2021-04-19 10905, 2021
ruaok
because there are other functions about redis we want to use and we'll constantly have to keep BU up to date.
2021-04-19 10934, 2021
alastairp
yeah, every time we need another datatype we have to add more methods which just call exactly the same redis method
2021-04-19 10939, 2021
iliekcomputers
\o/
2021-04-19 10947, 2021
iliekcomputers
Correct about what?
2021-04-19 10913, 2021
alastairp
does it matter? ;)
2021-04-19 10923, 2021
alastairp
ruaok: maybe we can grab some time on Weds to ensure that we go about this correctly
2021-04-19 10927, 2021
iliekcomputers
Lol
2021-04-19 10935, 2021
ruaok
k
2021-04-19 10944, 2021
alastairp
iliekcomputers: some time ago you suggested a "data" module in BU for storing long-term data in redis
People up for reviews: alastairp, ruaok, reosarevok, Freso, zas, yvanzo, Mr_Monkey, bitmap, _lucifer, CatQuest, shivam-kapila – anyone else who want to give review, let me know ASAP. :)
2021-04-19 10905, 2021
Freso
I haven’t received any mailed‐in reviews, so…
2021-04-19 10906, 2021
Freso
alastairp: Go!
2021-04-19 10910, 2021
alastairp
hello
2021-04-19 10938, 2021
alastairp
This week I worked with _lucifer to finish off the last breaking api changes in BU in order to release BU 2.0. This is mostly finished, pending a small discussion about redis
last Wednesday we found a bug in the listen processing pipeline that was causing listens to get stuck, we made a quick fix for the bug, but it openend up some more validation tasks that we have to do during listen reading
2021-04-19 10933, 2021
_lucifer
(I am a bit busy rn. can i please go last?)
2021-04-19 10914, 2021
alastairp
with Mr_Monkey we also looked into some other issues surrounding incoming listen validation (which fields are required and what values they can have) and how we can improve that in some cases
2021-04-19 10934, 2021
Mr_Monkey
The Strange Case of the Missing Track Title
2021-04-19 10935, 2021
alastairp
finally, I ran some tests to get AB dumps and incremental dumps working, that's ongoing
2021-04-19 10944, 2021
alastairp
Mr_Monkey: thanks for speaking up, you can go next
2021-04-19 10951, 2021
Mr_Monkey
Hello !
2021-04-19 10950, 2021
Mr_Monkey
Last week I worked on ListenBrainz listens pagination along with ruaok, and also had a look at why the Spotify player is currently not working on LB. Sigh, neither of the main players are working at the moment…
2021-04-19 10913, 2021
Mr_Monkey
Turns out the Spotify API is still in beta and they don't seem to announce changes
2021-04-19 10925, 2021
Mr_Monkey
So I'm expecting more of that until they hit a stable release
2021-04-19 10926, 2021
ruaok
oh, that is convenient!
2021-04-19 10949, 2021
Mr_Monkey
Also reviewed GSoC application drafts adn lots of PRs !
2021-04-19 10912, 2021
Mr_Monkey
Also fixed LB-864 real quick, that was an easy one for once
