#metabrainz

/

      • BrainzGit
        [brainzutils-python] amCap1712 merged pull request #65 (master…junit): Publish Unit Test Results https://github.com/metabrainz/brainzutils-pytho...
      • _lucifer
        alastairp, i think we have had this conversation but to confirm https://github.com/metabrainz/brainzutils-pytho... . we should replace `_prep_keys_list` with `_prep_key` here?
      • alastairp
        _lucifer: hmm, I don't recall having that conversation. But good point. not sure why we are using the list version in all of these methods that just take a single key as argument
      • _lucifer: https://github.com/metabrainz/listenbrainz-serv... did you also create .finish files for each of these? I don't see them in the branch
      • _lucifer
        alastairp: no i haven't. because the content will depend on how the common script to use. so i am waiting for the common script before i add the .finish files.
      • alastairp
        right. no problem. I'm on it
      • _lucifer
        ruaok: recs, similar users and stats all generated. a number of stats job will run again tonight so we'll know by tomorrow if there is some other issue with the cluster. otherwise its all ready.
      • ruaok
        sweet, thank you so much!
      • _lucifer: alastairp: am I doing this right?
      • why is there a lag after updating redis?
      • _lucifer
        i do not understand what you mean.
      • also can you check redis for which keys are present
      • ruaok
        I set/increment a value, and then I fetch it again right after and the value is not updated.
      • but when I call it again during the next test, it is updated from the previous run.
      • _lucifer
        interesting that's odd
      • ruaok
        oh. do I need to call something to finish that redis call?
      • _lucifer
        no increment just calls incrby so i don;t think so
      • ruaok
        its gotta be something else then
      • _lucifer
        *calls incr
      • sumedh joined the channel
      • alastairp
        ruaok: oh, mmm
      • we may have given you incomplete information
      • _lucifer
        can you check the keys inside redis? i have a hunch that there might be an issue with keys.
      • alastairp
        yeah, will need to manually prep keys if using _r
      • cache._r.incrby(cache._prep_key(REDIS_USER_LISTEN_COUNT + user_name, None), user_counts[user_name])
      • that prepends the global namespace to the keyname. so you were dealing with 2 different keys here
      • ruaok
        which would explain my issue. :)
      • lets see if test pass this time.
      • it ferreted out another problem, so not all bad.
      • and also I need to encode my own values, no?
      • _lucifer
        don't think so increment is not doing any encoding
      • alastairp
        ah yeah, this is the long discussion that _lucifer and I had last week
      • ruaok
      • alastairp
        counters aren't encoded, because we rely on redis to actually do the modification
      • feh, so you'll have to do `cache.get(whatever, decode=False)`
      • _lucifer: I still don't like that, even though we decided to go with it
      • _lucifer
        alastairp, we decided to add encoding but default to false, right?
      • alastairp
        I'll have to look at the chat logs again. I thought we were going to automatically encode/decode text values, and have the choice up to the user for the bulk get options
      • ruaok
        should `cache.set(REDIS_USER_LISTEN_COUNT + user_name, 0, time=0)`have encode=False then too?
      • alastairp
        I think I'll have to have a chat with ruaok about this on wednesday
      • ruaok: ah. so you're setting a value manually, then you want to incr, then you want to get?
      • ruaok
        the boom in the paste above.
      • alastairp
        yes, you'll need encode=False
      • so all of today's discussion makes it clear that this is super fragile and not at all intuitive
      • ruaok nods a lot
      • the context is: the cache is supposed to be magic and you're supposed to be able to go "cache x object" and it'll encode it to bytes, and then decode it seamlessly when you want the value back out
      • which works OK when you're just treating it as a kv store where you want to put strings, or integers
      • ruaok
        tasty magic
      • alastairp
        but now we're trying to actually use redis as redis
      • ruaok
        but gets muddled with inc
      • alastairp
        yeah, so things like inc, or hashes, or sets introduce additional complexity
      • oh, I just had an epiphany
      • we had this discussion about cache/data store split
      • what if we had the cache module, which is _just_ get/set, and does serialisation. and the data module which basically exposes redis, and doesn't do any serialisation
      • it's clear that ruaok's usage is data storage. he doesn't need any magic here (other than what redis provides)
      • ruaok
        yes, something like that.
      • alastairp
        so turns out that iliekcomputers was correct 3 years ago
      • !m iliekcomputers
      • BrainzBot
        You're doing good work, iliekcomputers!
      • ruaok
        because there are other functions about redis we want to use and we'll constantly have to keep BU up to date.
      • alastairp
        yeah, every time we need another datatype we have to add more methods which just call exactly the same redis method
      • iliekcomputers
        \o/
      • Correct about what?
      • alastairp
        does it matter? ;)
      • ruaok: maybe we can grab some time on Weds to ensure that we go about this correctly
      • iliekcomputers
        Lol
      • ruaok
        k
      • alastairp
        iliekcomputers: some time ago you suggested a "data" module in BU for storing long-term data in redis
      • ruaok
      • how do I get back to the int I wanted?
      • cast to int() from bytes?
      • alastairp
        yeah. cache module doesn't know that you're dealing with numbers here
      • cast to int is fine
      • reosarevok
        yvanzo, bitmap: releasing today, right?
      • bitmap
        reosarevok: yep, was there anything else we needed to fix on beta?
      • I only saw the recording checkbox one you fixed
      • alastairp
        _lucifer: so now I have no idea if this even works: https://github.com/metabrainz/brainzutils-pytho...
      • sumedh has quit
      • _lucifer
        i think it should be working. otherwise we would be seeing errors in sentry ?
      • alastairp
        yeah, probably. I don't see it using `self.key` anywhere other than that line anyway...
      • the version that LB uses doesn't use the cache module for this
      • yvanzo
        reosarevok: yes, I prepared the musicbrainz-docker release only.
      • reosarevok
        bitmap: AFAIK, that was it
      • We'll find out tomorrow how it was not, but :p
      • alastairp
        atj: hey, are you around? (and will you be around in approx 1hr?)
      • Freso
        <BANG>
      • It’s Bicycle Monday!
      • People up for reviews: alastairp, ruaok, reosarevok, Freso, zas, yvanzo, Mr_Monkey, bitmap, _lucifer, CatQuest, shivam-kapila – anyone else who want to give review, let me know ASAP. :)
      • I haven’t received any mailed‐in reviews, so…
      • alastairp: Go!
      • alastairp
        hello
      • This week I worked with _lucifer to finish off the last breaking api changes in BU in order to release BU 2.0. This is mostly finished, pending a small discussion about redis
      • I spent some time finishing off more detailed reporting during startup of LB and other python brainzes (https://github.com/metabrainz/listenbrainz-serv...)
      • last Wednesday we found a bug in the listen processing pipeline that was causing listens to get stuck, we made a quick fix for the bug, but it openend up some more validation tasks that we have to do during listen reading
      • _lucifer
        (I am a bit busy rn. can i please go last?)
      • alastairp
        with Mr_Monkey we also looked into some other issues surrounding incoming listen validation (which fields are required and what values they can have) and how we can improve that in some cases
      • Mr_Monkey
        The Strange Case of the Missing Track Title
      • alastairp
        finally, I ran some tests to get AB dumps and incremental dumps working, that's ongoing
      • Mr_Monkey: thanks for speaking up, you can go next
      • Mr_Monkey
        Hello !
      • Last week I worked on ListenBrainz listens pagination along with ruaok, and also had a look at why the Spotify player is currently not working on LB. Sigh, neither of the main players are working at the moment…
      • Turns out the Spotify API is still in beta and they don't seem to announce changes
      • So I'm expecting more of that until they hit a stable release
      • ruaok
        oh, that is convenient!
      • Mr_Monkey
        Also reviewed GSoC application drafts adn lots of PRs !
      • Also fixed LB-864 real quick, that was an easy one for once
      • BrainzBot
        LB-864: Newer/Older buttons do not work as links https://tickets.metabrainz.org/browse/LB-864
      • CatQuest
        :|
      • Mr_Monkey
        Also finished working on refactoring alert notification on the UI in LB
      • And looks like that's it for me ! reosarevok go !
      • CatQuest
        hmm
      • reosarevok
        Hi!
      • Mr_Monkey
        CatQuest: Not "for once" as in among the issues you post, just in general :p
      • Freso
        (People still up: ruaok, Freso, zas, yvanzo, bitmap, CatQuest, shivam-kapila, _lucifer – anyone else who want to give review, let me know ASAP.)
      • CatQuest
        Mr_Monkey: it wasn't that, see pm
      • reosarevok
        Last week I started some work on MBS-1658 (expect that to continue bit by bit over several weeks)
      • BrainzBot
        MBS-1658: My Collection: add free text comment field https://tickets.metabrainz.org/browse/MBS-1658
      • reosarevok
        Also started porting most of the report tables to react-table
      • Other than that, smaller issues and fixes, triaging, etc
      • Now releasing a new server
      • And I guess I did put a few new instrument images up :)
      • CatQuest: what about you?
      • CatQuest
        Meow! `😺`
      • not much, adding books, enjoying the weather
      • fin, go zas
      • zas
        hey
      • we released Picard 2.6.1
      • also fixed few things on new Jackson5 servers
      • upgraded discourse
      • I also deployed a nginx server to serve ftp files (on williams)
      • very simple, read-only container
      • with http(s)
      • (and http2)
      • that's it for me. ruaok ?
      • ruaok
        hey.
      • I spent most of the week trying to fix on issue in timescale listen fetching.
      • which begat a pile of wormholes and now I'm finishing a 30+ file PR on this. heh.
      • I just updated test.listenbrainz.org -- loading pages for the first time is going to be slow.
      • but once the caches are warm, it should start loading listens pages much much faster.
      • and I worked on various background tasks at MeB...
      • including working with a potential customer who "really wanted to sign last week" to instill urgency.
      • I work fast and then days an days nothing from them. heh.
      • well, hopefully 🦄 soon.