#metabrainz

/

      • yyoung
        Hi yvanzo, the 'v-2021-04-19' link to the blog is wrong, please fix it :) https://github.com/metabrainz/musicbrainz-docke...
      • flamingspinach has quit
      • flamingspinach joined the channel
      • sumedh joined the channel
      • Lotheric_ joined the channel
      • ZaphodBeeblebrox joined the channel
      • ZaphodBeeblebrox has quit
      • ZaphodBeeblebrox joined the channel
      • MRiddickW_ joined the channel
      • Lotheric has quit
      • MRiddickW has quit
      • CatQuest has quit
      • sumedh has quit
      • sumedh joined the channel
      • yvanzo
        Thanks, fixed.
      • BrainzGit
        [musicbrainz-server] reosarevok opened pull request #2069 (master…MBS-11598): MBS-11598: Do not generate image link if we don't have a suffix https://github.com/metabrainz/musicbrainz-serve...
      • Etua joined the channel
      • Etua has quit
      • sampsyo has quit
      • sampsyo joined the channel
      • MRiddickW_ has quit
      • Darkloke joined the channel
      • _lucifer
      • alastairp: ruaok: does the above query look fine? it works in psql, i am wondering if sqlalchemy will correctly quote the `:error_message` param.
      • ruaok
        not sure about that one.
      • _lucifer
        for example, if i pass `error_message="Hello, World!"` i want that `:error_message` be substituted with `'"Hello, World!"'`
      • alastairp
        my feeling is that :error_message should get quoted properly
      • but I couldn't tell you for sure, probably better to just load up a shell and try it
      • _lucifer
        yeah, i'll do that.
      • alastairp
        service_details = jsonb_set(coalesce(service_details, '{}'), '{error_message}', 'hello world')
      • is this what you want?
      • I don't think it'll double-quote it, not sure if that's what you're asking for
      • _lucifer
        yeah, the double quotes are needed in this case
      • alastairp
        I suspect you'll have to pass in {"error_message": '"' + message + '"'} when passing args to execute, if it lets you do it
      • or maybe ":error_message" in coalesce, but again, unsure sorry
      • try a few things and tell us which one works :)
      • _lucifer
        sure :D
      • BrainzGit
        [listenbrainz-server] MonkeyDo opened pull request #1395 (master…monkey-fix-spotify-player): Spotify player fixes and improvements https://github.com/metabrainz/listenbrainz-serv...
      • Darkloke has quit
      • Lotheric_ is now known as Lotheric
      • Sophist_UK joined the channel
      • ruaok
        yvanzo: any idea what is up with the sir queues?
      • Sophist-UK has quit
      • chaban
        The last few days I've been marking thousands of recordings as video. Could it be related?
      • MRiddickW joined the channel
      • yvanzo
        chaban: that seems unlikely to be related, unless you have peaks of high activity.
      • ruaok: not yet, there is no error from sir itself at least.
      • ruaok
        Mr_Monkey: remind me please, which two time_range tickets are fixed by my work?
      • Mr_Monkey
        Let me fish those out
      • LB-701 and LB-862
      • BrainzBot
        LB-701: My listen page is almost empty https://tickets.metabrainz.org/browse/LB-701
      • LB-862: Listens view broken after deleting last.fm imported tracks https://tickets.metabrainz.org/browse/LB-862
      • ruaok
        thx
      • _lucifer
        alastairp: ruaok: json.dumps works.
      • ruaok
        what exactly?
      • _lucifer
      • ruaok
        ah, great.
      • _lucifer
        ah, just doing '"{}"'.format(error_message) also works. the previous time i tried that i had made a typo due to which it hadn't worked.
      • ruaok
        _lucifer: alastairp: https://github.com/metabrainz/listenbrainz-serv... is ready for review.
      • alastairp
        ok, thanks
      • _lucifer
        👍
      • alastairp
        _lucifer: is error_message a string, or a dictionary?
      • _lucifer
        string
      • alastairp
        so in this case you're only using json.dumps to add quotes to it?
      • _lucifer
        yes, i was. but format thing worked as well so i'll go ahead with it,
      • alastairp
        I've done this before where I use json.JSONEncoder().encode(val)
      • because, consider:
      • s = 'foo"bar'; json.dumps(s)
      • '"foo\\"bar"'
      • _lucifer
        ah! makes sense.
      • ruaok
        alastairp: got a sec for a sanity check?
      • alastairp
        give me 5, just fighting with haproxy
      • ruaok
        perfect. it will take me 5 minutes to explain. ;D
      • _lucifer
        is there any difference between using dumps or the encoder?
      • ruaok
        the listen count are very expensive to calculate and when we first push this branch into production, the site will appear effectively broken because nothing is loading.
      • to mitigate that, I plan to write a script that:
      • 1) Picks the last inserted timestamp from the listen table.
      • 2) For each user, set a zero listen count, zero timestamp.
      • 3) iterate over every listen from begin of time to this timestamp. tabulate listen counts and time.
      • _lucifer
        how much expensive are we talking about?
      • ruaok
        4) At the end of this script INCREMENT each of the users counts by the calculated total. Update timestamps to ensure that calculated timestamps won't overwrite any timestamps that may have been calculated since this process started.
      • This process allows for the timescale_writer to keep writing new listens and for the update script and the timescale writer to coexist peacefully.
      • In theory we should catch all the listens as they come in.
      • thoughts?
      • _lucifer: I saw one take 85s.
      • _lucifer
        😵
      • alastairp
        here
      • ruaok
        exactly, which is why I really want to stop computing them.
      • alastairp
        yeah, doing a script to pre-compute these things makes a lot of sense. does this mean we'll have to make a deploy with the infrastructure available but disabled? or will you be able to do it before the big deploy?
      • tabulate listen counts and time - this will just find min/max time and num listens for each user?
      • ruaok
        > tabulate listen counts and time - this will just find min/max time and num listens for each user?
      • yes
      • > or will you be able to do it before the big deploy?
      • I think that if we deploy the timescale_writer first, then we can run the tabulate script, then push the web container, then I think the right thing will happen.
      • More correctly:
      • 1. tabulate.
      • 2. deploy timescale writer
      • 3. tabulate again
      • 4. deploy web
      • then the release should be seamless.
      • alastairp
        what service updates max and num listens when necessary? timescale writer?
      • ruaok
        was timescale writer, but a lot of that logic has moved to timescale listenstore.
      • where it belongs.
      • alastairp
        but the writer container, not the web container?
      • ruaok
        there is a lot of really needed cleanup in this PR.
      • alastairp
        do you need any special code running in a container in order to do 1 ?
      • ruaok
        yes,its runs in the writer container with no special code needed.
      • _lucifer
        beta, and prod share the cache container so we could run the script in beta if needed?
      • ruaok
        just python module speaking that logic moved to the listenstore.
      • _lucifer: yeah, sure. good idea.
      • alastairp
        ruaok: yeah, that's what I was trying to get at. running in beta should be fine
      • ruaok
        the key being, the writer needs to be the first thing to move as part of the deployment.
      • you can test all this one test.lb right now.
      • load your feed page to find out what I mean.
      • it will appear broken, I assure you.
      • alastairp
        what's the purpose of 3? because of stuff that might come in between tabulate and when we shut down the old timescale writer?
      • ruaok
        #1 is needed for #2 to work right. #3 is the part that does the actual work.
      • I suppose #1 could be reduced to "set all redis keys to 0."
      • but that is more code to write, lol
      • alastairp
        ah, I follow now
      • _lucifer
        another thing, how about getting the list of all users and then querying the api in beta container for all users.
      • instead of writing another script for the same task.
      • ruaok
        that could work, but it would tie up the DB for hours. and it would make N passes over the data, as opposed to 1.
      • and this other script I am talking about, all its pieces already exist.
      • its just a matter of conjuring them into one script.
      • _lucifer
        ah ok! a special script makes sense then
      • ruaok signs the 🦄's butt
      • ruaok
        er, contract.
      • _lucifer
        lol
      • shivam-kapila
        Lol. I feel ya. Fiddling with docs too
      • ruaok
        lol. :)
      • no, the contract hasn't been countersigned yet.
      • alastairp
        ok hey
      • we got part of https://freesound.org/ running in a kubernetes cluster
      • there's definitely something to be said for just sshing into a computer and running your webserver in a screen
      • _lucifer
        nice but why the move to kubes?
      • alastairp
        because our IT department supports it
      • _lucifer
        ah! :D
      • yvanzo
        I would love to read more your experience about it :)
      • alastairp
        🤮
      • lol
      • nah, it's not too bad. we're lucky that IT gave us a set of templates to copy and fill out.
      • yvanzo
        :D
      • alastairp
        there are a lot of moving parts, and coming into it with no knowledge about how everything fits together there was a lot of guessing. I'm sure that as we migrate more services to it we'll come to understand better how everything works
      • some things like auto-scaling are really neat
      • _lucifer
      • ruaok
        someone has an ego problem, for sure.
      • iliekcomputers
        big tech and personal project things -- name a worse pair
      • ruaok
        crocs and high heels?