#metabrainz

/

      • thomasross has quit
      • MajorLurker joined the channel
      • ruaok
        moooin!
      • look what was released ON THE DAY we upgraded to timescale 2.2.0: https://blog.timescale.com/blog/how-we-made-dis...
      • this exact release provides the feature we most needed. skip-indexes.
      • sigh. well, once we release things will be faster. and once we upgrade to 2.2.1 things will get even faster.
      • _lucifer
        we can upgrade again, this one should be simpler
      • ruaok
        which provides a nice basis for us to scale the service.
      • _lucifer: agreed, but I think we should go pg12 as well this time.
      • but first I want to get time-ranges merged and deployed.
      • _lucifer
        yeah makes sense to do that.
      • ruaok
        Mr_Monkey: we have a failing test for the frontend on the time-ranges PR. I'm hoping to merge and deploy this today: https://github.com/metabrainz/listenbrainz-serv...
      • can you please have a look?
      • _lucifer
        that's the phantom bug Mr_Monkey mentioned yesterday methinks.
      • it goes away on upgrading node.
      • alastairp
        ruaok: amazing. glad to see that there's continual improvements going on with this work
      • ruaok
        _lucifer: so, we need to upgrade node in order to release this PR??
      • gah.
      • ruaok rages against javascript
      • alastairp
        I'm just finishing some writing now. I've got a meeting at 12, so I'll start reviewing during it (sigh. mandatory meetings)
      • _lucifer
        no, my understanding is that this issue only occurs in tests.
      • so we could mark the test as skipped for now and visit it again when we upgrade node.
      • alastairp
        if it works in prod I'm happy to merge despite a failing test
      • ruaok
        alastairp: ok, but lets really stay on point with this. listen counts are no longer being updated in production because of the crap disk space issues.
      • we really need to get this out today.
      • #1396 also needs to go out at the same time.
      • _lucifer: there is a failing spark test as well: https://github.com/metabrainz/listenbrainz-serv...
      • any ideas on that?
      • _lucifer
        seems the build failed while trying to install python
      • i'll rerun the jobs and see if that goes away
      • Mr_Monkey
        ruaok: Yes, as mentionned I need to update Node version, but I started that last night.
      • Alternatively, you can merge as-is, I sexuple-checked that the mechanism works correctly, and using a more recent Node version all tests pass.
      • ruaok
        Mr_Monkey: great, thanks for verifying.
      • Strezless joined the channel
      • sumedh joined the channel
      • _lucifer
        ruaok, i want to test some further improvements for the spark request consumer. will need to take down/up the container few times and issue some requests. is that fine?
      • Strezless has quit
      • ruaok
        Sure!
      • _lucifer
        thanks!
      • BrainzGit
        [musicbrainz-server] reosarevok opened pull request #2094 (master…MBS-960): MBS-960: Allow mass-deleting and mass-changing user tags https://github.com/metabrainz/musicbrainz-serve...
      • _lucifer
        ruaok, what's the right way to run a script in the background? i found nohup, setsid or & but not sure which to use?
      • ruaok
        script.py &
      • will run in the background, but if you close that terminal, the process gets killed.
      • nuhup script.py &
      • will continue running in the background.
      • nohup script.py 2>&1 > <file> &
      • should capture output into a file and keep running if the terminal disconnects.
      • _lucifer
        yup that the last command is what i am exactly looking for. thanks!
      • ruaok
        likely to only work in bash.
      • alias m='make 2>&1 | less'
      • is an alias I have for make'ing C programs.
      • _lucifer
        ruaok, driver running on newleader without docker now :D
      • and python deps also managed inside a venv so no issues there.
      • alastairp, after our discussion yesterday, i revisted the docs. i had misunderstood those, the deps do get distributed to the driver at least in the yarn client mode which is the one we use. thanks!
      • ruaok
        nice. remind me, what was the motivation for getting rid of docker?
      • _lucifer
        doesn't add anything and requires additional setup. right we have to build the driver image and update the git repo on newleader to ensure things work correctly.
      • now its just update the repo on newleader and execute the startup script.
      • alastairp
        ruaok: I did a preliminary review of the PR, but skipped the listenstore file for now in order to get some comments to you. Looking at listenstore now
      • ruaok
        alastairp: great, thanks.
      • _lucifer: ahh, I see. I think we may want to make the request_consumer into an actual daemon, rather than starting with with nohup.
      • alastairp
        unit tests are passing in github, but failing for me (`self.listenstore.insert(listens)` -> AttributeError: 'NoneType' object has no attribute 'insert'), looking into why
      • ruaok
        I'll get moving on the other comments.
      • _lucifer
        ruaok, sounds interesting. but i do not know how to do that. let's discuss it in detail later when you are available.
      • ruaok
        +1
      • alastairp
        _lucifer: in fact, the app startup process inside our docker (service files, etc) is exactly the same process as this, so we could reuse the mechanism
      • _lucifer
        that would be even better :D
      • BrainzGit
        [listenbrainz-server] alastair opened pull request #1433 (master…spelling): Fix grammar https://github.com/metabrainz/listenbrainz-serv...
      • alastairp
        ruaok: do we have a small data dump that we can import for local development/testing?
      • ruaok
        no
      • alastairp
        ruaok: we can improve cache init in LB-879
      • BrainzBot
        LB-879: unify external service connections in single location https://tickets.metabrainz.org/browse/LB-879
      • ruaok
        k
      • alastairp
      • me> wtf, why are all of my listens in the test database duplicated?
      • also me> oh that's right, I'm listening to my playlist of covers
      • BrainzGit
        [musicbrainz-server] reosarevok opened pull request #2095 (master…MBS-11644): MBS-11644: Allow specifying a release MBID when moving a CD TOC https://github.com/metabrainz/musicbrainz-serve...
      • reosarevok
        Sigh, more fucking JIRA spam
      • I thought I blocked this asshole
      • I'm fixing it - lots of crap on AB
      • alastairp
        thanks reosarevok
      • reosarevok
        alastairp: is https://tickets.metabrainz.org/browse/AB-409 meant to be fixed or not?
      • BrainzBot
        AB-409: Use bulk get methods for single recording endpoints
      • LordSputnik_ joined the channel
      • Mr_Monkey
        Freso: Bumping ROpdebee 's comment since it was easy to miss: This Jira user is spamming and possibly trying some SQL injection: https://tickets.metabrainz.org/secure/ViewProfi...
      • LordSputnik_ is now known as BenOckmore
      • Oh, just seeing your comment reosarevok are you on this?
      • reosarevok
        Mr_Monkey (and Freso): I blocked their jira editing privs (if I'm right about how that works) because just deleting the user makes it so that they can re-add it immediately
      • alastairp
        reosarevok: it looks like it's not fixed yet. leave open (or review submittedn) thanks
      • reosarevok
        But this is the second time they do this
      • So, we could use a better way
      • They also tried to spam our github at some point
      • Mr_Monkey
        Thanks for dealing with it. Can't imagine what they're trying to get out of that.
      • ruaok
        I dont understand most spammers.
      • reosarevok
        Chances are this specific one just has some sort of mental issue? The whole thing is super rambling. But who knows
      • Anyway, if you see it happen again, please do ping me, Freso or yvanzo ASAP
      • Any of us should be able to deal with it
      • alastairp
        Mr_Monkey: I'm seeing some weird frontend interaction issues with the new time ranges PR
      • after loading some listens from spotify. I'm looking into it a bit more, but I'm not sure if it's due to the time range storage/redis, or something in the frontend code
      • (local development version)
      • ruaok
        try deleting the redis volume and restarting.
      • alastairp
        trying that
      • ruaok
        first bits of feedback fixes pushed.
      • alastairp
        that doesn't seem to be it - I think it's mostly to do with the older/oldest newer/newest buttons. it's making the queries to get the updated data, but doesn't seem to be displaying it
      • ruaok
        Mr_Monkey: alastairp: https://vacunacovid.catsalut.gencat.cat/ is now starting to have useful info. Even though I fit the age bracket that is being vaccinated now, I tried for a cita and got a: Your group is hasn't been called yet. I'll keep trying.
      • alastairp
        I'm testing the api queries individually now
      • ruaok: cool, thanks (to both)
      • ruaok
        any comments on the listenstore yet?
      • if you have comments pending, post them please.
      • ruaok is idle
      • _lucifer: not that is sounds relevant anymore, but a proper linux daemon is created as such: https://stackoverflow.com/questions/17954432/cr...
      • _lucifer
        thanks! i'll take a look.
      • alastairp
        ruaok: this is what I'm trying to work out with the frontend - now that I see some problems with the frontend I'll try and use the API to test the listenstore independently
      • ruaok
        what problems are you seeing? anything I can help with?
      • alastairp
      • this is the data that I have in my local database
      • and redis
      • I haven't run anything to generate user counts in redis (lc.[username] key and lc-total? )
      • I started from an empty database and empty redis volume, created the db, logged in, and linked spotify
      • ruaok
        I dont see a problem so far.
      • alastairp
        listen_count_30day has a single row for me, says that count is 46. that looks good
      • when is the listenbrainz:lc.alastairp key created?
      • ruaok
        when get timestamps_per_user() is called.
      • alastairp
        this is from localhost. I don't know where (18611 total) is coming from here. coincidenally this is also the number displayed on listenbrainz.org: https://listenbrainz.org/user/alastairp
      • ruaok
        er no.
      • get_listen_count_for_user
      • _lucifer
        all along the watchtower is nice song :)
      • ruaok
        there is an API call to fetch that.
      • alastairp
        I see it
      • ruaok
        /user/<user_name>/listen-count
      • call that and check the redis keys again
      • alastairp
        yep, the key is there now
      • _lucifer
        you might need to configure the API_URL in config.py
      • ruaok
        ok, then it really is the front end.
      • _lucifer
        by default it queries lb.org not the local
      • ruaok
        _lucifer: ohh, good one.
      • alastairp
        ahhhh - that'll be it. localhost dev will be hitting lb.org
      • ruaok
        that would do it.
      • alastairp
        thanks
      • ruaok
        phew.
      • ruaok runs out to fetch some noms. back in a jiffy
      • alastairp
        > Recent listens (46 total)
      • phew
      • oh yeah, API works fine now
      • frontend does everything properly
      • Mr_Monkey
        Yay ! 🎉
      • ruaok
        _lucifer: I dont understand your last two questions on that PR "would this be... " and "I was thinking... "
      • well you and alastairp, really. is there an action for me?
      • _lucifer
        i meant something on the line of HSET listenbrainz.ts:rob min 0 max 100 and then HGET listenbrainz.ts:rob min or HGETALL