#metabrainz

/

      • ruaok
        np.
      • 2021-04-19 10903, 2021

      • _lucifer
        happens to me all the time :)
      • 2021-04-19 10907, 2021

      • alastairp
        _lucifer: looks good. don't forget to remove the assert false
      • 2021-04-19 10915, 2021

      • _lucifer
        yes
      • 2021-04-19 10932, 2021

      • alastairp
        is this an issue with other tests too?
      • 2021-04-19 10946, 2021

      • _lucifer
        yeah, all workflows.
      • 2021-04-19 10949, 2021

      • alastairp
        I'm planning to be in the office on Wednesday. should we aim for a PR cleanup?
      • 2021-04-19 10917, 2021

      • ruaok
        that would very good.
      • 2021-04-19 10902, 2021

      • alastairp
      • 2021-04-19 10924, 2021

      • alastairp
        gotta to be a good feeling for the people who work there
      • 2021-04-19 10907, 2021

      • _lucifer
        alastairp, also i forgot to mention that the tag workflow didn't work :(. i'll debug why so.
      • 2021-04-19 10930, 2021

      • alastairp
        oh, that's sad. let me know what you find
      • 2021-04-19 10945, 2021

      • _lucifer
        sure.
      • 2021-04-19 10938, 2021

      • BrainzGit
        [listenbrainz-server] amCap1712 merged pull request #1393 (master…gh): Check for CI env variable https://github.com/metabrainz/listenbrainz-server…
      • 2021-04-19 10934, 2021

      • MRiddickW joined the channel
      • 2021-04-19 10911, 2021

      • ruaok
        alastairp: fetching min/max timestamps using the cont agg takes nearly 3s. doing it directly from the listens table? 1.1s
      • 2021-04-19 10958, 2021

      • ShivamAwasthi joined the channel
      • 2021-04-19 10910, 2021

      • ShivamAwasthi has quit
      • 2021-04-19 10935, 2021

      • alastairp
        mmm, odd
      • 2021-04-19 10944, 2021

      • alastairp
        this is past my postgres knowledge
      • 2021-04-19 10910, 2021

      • ruaok
        the thing to know: continous aggregates are pretty damn slow.
      • 2021-04-19 10959, 2021

      • ruaok
        514k rows in this view. pretty sad.
      • 2021-04-19 10949, 2021

      • ruaok
        hmmm. let me ensure that indexes a built for the view
      • 2021-04-19 10916, 2021

      • ruaok
        oh wow.
      • 2021-04-19 10918, 2021

      • ruaok
      • 2021-04-19 10935, 2021

      • ruaok
        ok, some users are fast to fetch directly. others, not so.
      • 2021-04-19 10931, 2021

      • _lucifer
        ruaok: your tests are now successfully failing \o/ lol :D
      • 2021-04-19 10942, 2021

      • ruaok
        lolol. very good, thanks.
      • 2021-04-19 10906, 2021

      • ruaok
        quick question: the `cache.get(REDIS_USER_TIMESTAMPS + user_name, decode=False)`
      • 2021-04-19 10932, 2021

      • _lucifer
        you need the namespace as well there if you passed one during set
      • 2021-04-19 10933, 2021

      • ruaok
        call returns bytes... previously we were using encoding and it returned str()
      • 2021-04-19 10949, 2021

      • ruaok
      • 2021-04-19 10918, 2021

      • ruaok
        the namespace is set when the timescale listenstore starts.
      • 2021-04-19 10947, 2021

      • ruaok
        so, my keys have namespaces set -- should be ok, no?
      • 2021-04-19 10908, 2021

      • _lucifer
        let me check and get back that to you
      • 2021-04-19 10919, 2021

      • _lucifer
        regarding str pass decode=True
      • 2021-04-19 10927, 2021

      • _lucifer
        what type do you want bytes or str?
      • 2021-04-19 10932, 2021

      • ruaok
        str
      • 2021-04-19 10944, 2021

      • ruaok
        but I don't really want me keys to be encoded. there is no need.
      • 2021-04-19 10950, 2021

      • _lucifer
        yes, then pass encode=True and decode=True
      • 2021-04-19 10957, 2021

      • alastairp
        encode/decode is for the value, not the key
      • 2021-04-19 10901, 2021

      • alastairp
        we don't encode keys any more
      • 2021-04-19 10919, 2021

      • ruaok
        understood, for the value.
      • 2021-04-19 10923, 2021

      • _lucifer
        that change will come in LB once we upgrade BU
      • 2021-04-19 10937, 2021

      • _lucifer
        (not encoding keys)
      • 2021-04-19 10953, 2021

      • ruaok is really confused now.
      • 2021-04-19 10913, 2021

      • alastairp
        maybe we're talking about a few different things
      • 2021-04-19 10922, 2021

      • ruaok
        can one of you please tell me the correct and future proof way to set/get keys?
      • 2021-04-19 10941, 2021

      • ruaok
        ideally no encoded that we don't have to rescan the DB when new BU is deployed.
      • 2021-04-19 10945, 2021

      • _lucifer
        pass both encode and decode=True
      • 2021-04-19 10901, 2021

      • ruaok
        and that will currently encode my values?
      • 2021-04-19 10919, 2021

      • ruaok
        sot that will require a flush of all the values when the new BU comes?
      • 2021-04-19 10921, 2021

      • _lucifer
        yes that will correctly encode the values.
      • 2021-04-19 10927, 2021

      • ruaok
        (which is EXPENSIVE!)
      • 2021-04-19 10901, 2021

      • _lucifer
        i am not sure regarding the flush because the change is to encoding of keys not values. maybe alastairp can tell about that.
      • 2021-04-19 10911, 2021

      • alastairp
        no, you won't need to do anything when a new BU comes
      • 2021-04-19 10944, 2021

      • ruaok
        wow, this convo keeps getting worse. ;
      • 2021-04-19 10955, 2021

      • _lucifer
        we should prbably do the new BU upgrade soon. to avoid such confusions.
      • 2021-04-19 10920, 2021

      • alastairp
        the endode/decode flags are to take whatever value you pass into cache.set() and serialise it using msgpack. this means that you get out exactly what you get in
      • 2021-04-19 10938, 2021

      • alastairp
        this is the default and the cache has always worked like this
      • 2021-04-19 10910, 2021

      • alastairp
        we're not changing this behaviour in the upgrades to BU
      • 2021-04-19 10912, 2021

      • ruaok
        yes, but does one return str() and the other bytes() ?
      • 2021-04-19 10905, 2021

      • alastairp
        yes, because the redis python library returns bytes. If you don't ask it to decode the value, you get the bytes back
      • 2021-04-19 10925, 2021

      • alastairp
        get(..., decode=True) returns a string, get(..., decode=False) return bytes
      • 2021-04-19 10937, 2021

      • ruaok
        so, when the decode/encode flags go away we will always return bytes?
      • 2021-04-19 10952, 2021

      • alastairp
        sorry, maybe there was a confusion here
      • 2021-04-19 10957, 2021

      • alastairp
        decode/encode are not going away
      • 2021-04-19 10931, 2021

      • ruaok
        ahhh, ok.
      • 2021-04-19 10917, 2021

      • yvanzo joined the channel
      • 2021-04-19 10927, 2021

      • BrainzGit
        [listenbrainz-server] amCap1712 opened pull request #1394 (master…gh-tag): Test docker image workflow https://github.com/metabrainz/listenbrainz-server…
      • 2021-04-19 10954, 2021

      • _lucifer
        alastairp: https://github.com/metabrainz/listenbrainz-server… i removed the tag pattern and it seems to work. so i think due to some reason the pattern is not matching.
      • 2021-04-19 10944, 2021

      • _lucifer
        which is strange because i tried in regex101 online and the tag matches the pattern.
      • 2021-04-19 10953, 2021

      • alastairp
        _lucifer: I believe that these aren't regular expressions
      • 2021-04-19 10911, 2021

      • alastairp
      • 2021-04-19 10919, 2021

      • alastairp
        see that they're called "filter patterns"
      • 2021-04-19 10934, 2021

      • alastairp
        it's not clear that it supports `{4}`
      • 2021-04-19 10947, 2021

      • _lucifer
        yeah this is probably some adhoc github thing..
      • 2021-04-19 10909, 2021

      • _lucifer
      • 2021-04-19 10916, 2021

      • _lucifer
        had to remove the quotes
      • 2021-04-19 10952, 2021

      • _lucifer
        so maybe only use quotes if there is a * involved
      • 2021-04-19 10939, 2021

      • alastairp
        great!
      • 2021-04-19 10927, 2021

      • alastairp
        does this workflow use the image cache?
      • 2021-04-19 10934, 2021

      • _lucifer
        no
      • 2021-04-19 10958, 2021

      • _lucifer
        i did that intentionally so that its always a clean slate
      • 2021-04-19 10929, 2021

      • alastairp
        mmm, right.
      • 2021-04-19 10903, 2021

      • alastairp
        see that it's up to 7 minutes already, though. I don't really want to stick around waiting this long for it to build every time
      • 2021-04-19 10938, 2021

      • alastairp
        especially if we need to do quick bugfix
      • 2021-04-19 10941, 2021

      • _lucifer
        yeah, that's a tradeoff true and if we build locally we do use the cache anyways i think.
      • 2021-04-19 10956, 2021

      • _lucifer
        so we could add image cache here as well then?
      • 2021-04-19 10931, 2021

      • alastairp
        the cache action actually caches docker images, rather than anything on the filesystem
      • 2021-04-19 10935, 2021

      • alastairp
        so I think it's OK
      • 2021-04-19 10952, 2021

      • alastairp
        one thing I don't know is how much space this is going to end up taking, and how much of our storage we're using
      • 2021-04-19 10906, 2021

      • alastairp
      • 2021-04-19 10930, 2021

      • alastairp
        two more cache items to review before we release
      • 2021-04-19 10944, 2021

      • _lucifer
      • 2021-04-19 10900, 2021

      • _lucifer
        maybe ruaok can take a look and see if we can find out how much we are using
      • 2021-04-19 10901, 2021

      • _lucifer
        but if it tops 5g for the repo it'll just evict old cache items.
      • 2021-04-19 10940, 2021

      • alastairp
        yeah, that's what I hope
      • 2021-04-19 10941, 2021

      • _lucifer
        alastairp, nice :D. i'll check and update those PRs
      • 2021-04-19 10911, 2021

      • sumedh joined the channel
      • 2021-04-19 10917, 2021

      • _lucifer
      • 2021-04-19 10908, 2021

      • ruaok
      • 2021-04-19 10919, 2021

      • ruaok
        _lucifer: our usage report for the last 30 days
      • 2021-04-19 10935, 2021

      • _lucifer
        thanks! :D
      • 2021-04-19 10952, 2021

      • _lucifer
        https://github.com/metabrainz/brainzutils-python/… this is the output of the junit action, alastairp
      • 2021-04-19 10907, 2021

      • _lucifer
        very strange, the usage report has nothing on BU or LB.
      • 2021-04-19 10911, 2021

      • alastairp
        I wonder if there's a different report
      • 2021-04-19 10920, 2021

      • alastairp
        this seems to be repo storage, not actions storage
      • 2021-04-19 10924, 2021

      • alastairp
        (maybe?)
      • 2021-04-19 10908, 2021

      • ruaok
      • 2021-04-19 10923, 2021

      • ruaok
        the report I shared cam from "get usage report"
      • 2021-04-19 10906, 2021

      • sumedh has quit
      • 2021-04-19 10943, 2021

      • alastairp
        thanks. "github actions" shows 0 because it seems that it only tracks actions on private repos (public repos are free). Based on their docs I'd expect "Storage for actions and packages" to show our cache usage
      • 2021-04-19 10914, 2021

      • alastairp
        but if that shows 0 then that's fine
      • 2021-04-19 10952, 2021

      • alastairp
        _lucifer: I like that output!
      • 2021-04-19 10948, 2021

      • ruaok
        I see that brainzutils cache increment does not have the ability to increment by more than 1.
      • 2021-04-19 10932, 2021

      • ruaok
        I need to do that, however. should I just make multiple calls or set a new value?
      • 2021-04-19 10906, 2021

      • ruaok
        in theory the latter can cause a race condition, but we only have one piece of code that inserts new listens right now, so it should never happen.
      • 2021-04-19 10911, 2021

      • ruaok
        thoughts, alastairp, _lucifer ?
      • 2021-04-19 10955, 2021

      • alastairp
        mm
      • 2021-04-19 10959, 2021

      • alastairp
        ruaok: which method?
      • 2021-04-19 10907, 2021

      • ruaok
        increment
      • 2021-04-19 10932, 2021

      • alastairp
        I think that's a bug
      • 2021-04-19 10937, 2021

      • alastairp
        we should expose INCRBY
      • 2021-04-19 10900, 2021

      • ruaok
        that would be ideal
      • 2021-04-19 10904, 2021

      • alastairp
        for now, can you either loop through, or manually call cache._r.incrby?
      • 2021-04-19 10911, 2021

      • alastairp
        and we'll have a version later in the week
      • 2021-04-19 10918, 2021

      • ruaok
        I'll do the latter.
      • 2021-04-19 10959, 2021

      • BrainzGit
        [brainzutils-python] alastair opened pull request #66 (master…incr-amount): Allow increment to take an optional amount value https://github.com/metabrainz/brainzutils-python/…
      • 2021-04-19 10928, 2021

      • alastairp
      • 2021-04-19 10906, 2021

      • alastairp
        ruaok: we're probably at least another day away from releasing BU 2.0, so if you need it now, _r.incr is still the best way to go for now
      • 2021-04-19 10918, 2021

      • ruaok
        k
      • 2021-04-19 10952, 2021

      • _lucifer
        alastairp: nice, so should we merge that junit action PR or keep it open for now?
      • 2021-04-19 10906, 2021

      • alastairp
        _lucifer: let's merge it
      • 2021-04-19 10915, 2021

      • _lucifer
        there are some other available as well https://github.com/marketplace?type=actions&q… if we want to take a look
      • 2021-04-19 10931, 2021

      • alastairp
        😓
      • 2021-04-19 10944, 2021

      • _lucifer
        regarding having this in LB, we can reduce 4 comments to 1 by running all tests as different jobs in the same workflow
      • 2021-04-19 10910, 2021

      • alastairp
        I don't really want to look into it more. do a github search and see which one is used most?
      • 2021-04-19 10925, 2021

      • alastairp
        also, check for commit activity maybe?
      • 2021-04-19 10932, 2021

      • alastairp
        oh yeah, that's a good idea for a future step
      • 2021-04-19 10932, 2021

      • _lucifer
        👍
      • 2021-04-19 10905, 2021

      • Etua joined the channel
      • 2021-04-19 10905, 2021

      • alastairp
        _lucifer: looking through that list, many of them are false positives, I think there are only 2 or 3 which actually do what we need
      • 2021-04-19 10934, 2021

      • _lucifer
        yeah, the one we are already using is by far the most used.
      • 2021-04-19 10947, 2021

      • alastairp
        cool. that's good enough for me
      • 2021-04-19 10901, 2021

      • ruaok
        the latest docker says 'Docker Compose is now in the Docker CLI, try `docker compose up`'
      • 2021-04-19 10913, 2021

      • ruaok
        finally, no need to install that as separate step.
      • 2021-04-19 10923, 2021

      • alastairp
        neat
      • 2021-04-19 10937, 2021

      • alastairp
        gonna be hell on my muscle memory
      • 2021-04-19 10917, 2021

      • Etua has quit