#metabrainz

/

      • mayhem
        ok, then I'll start a run with a wholly new sets of command line paramters.
      • 2022-04-14 10451, 2022

      • lucifer
        👍
      • 2022-04-14 10416, 2022

      • BrainzGit
        [musicbrainz-server] 14reosarevok opened pull request #2490 (03master…alias-edit-form-hydrate-div): Fix the hydrate div name for AliasEditForm https://github.com/metabrainz/musicbrainz-server/…
      • 2022-04-14 10427, 2022

      • reosarevok
        bitmap: remind me, this shouldn't cause issues, right? ^
      • 2022-04-14 10405, 2022

      • lucifer
        mayhem: requested anything?
      • 2022-04-14 10421, 2022

      • mayhem
        not yet, just getting going on that.
      • 2022-04-14 10427, 2022

      • lucifer
        ah ok 👍
      • 2022-04-14 10433, 2022

      • mayhem
        working on adding an nginx container for hosting the html files.
      • 2022-04-14 10435, 2022

      • lucifer
        btw if you want to read on the implicit feedback thing, the paper linked in the doc is a good read. ignore the maths part but the qualitative description is nice and describes many things which we can take inspiration from.
      • 2022-04-14 10437, 2022

      • lucifer
        ah ok 👍
      • 2022-04-14 10400, 2022

      • mayhem
        ah yes, thanks for the reminder.
      • 2022-04-14 10413, 2022

      • Ansh
        alastairp monkey : Thank you so much for your feedback.
      • 2022-04-14 10414, 2022

      • Ansh
        To show the reviews on the BB website, I tried a few UI mockups, but I think it would require some more discussion for the perfect placement.
      • 2022-04-14 10444, 2022

      • Ansh
        For OAuth login, I have thought of something similar to how we do in BB to authenticate with MB. I'll add some more information on the proposal.
      • 2022-04-14 10402, 2022

      • Ansh
      • 2022-04-14 10444, 2022

      • monkey
        Yes, i think you are correct in using a passport strategy, just wanted to have an overview of how it'll work.
      • 2022-04-14 10421, 2022

      • monkey
        Same for the reviews UI, I don't think we need a finalized mockup, but it's always nice to have some rough ideas on the table, even that early.
      • 2022-04-14 10439, 2022

      • mayhem
      • 2022-04-14 10421, 2022

      • mayhem
        I'm trying to specify a list of ranks on the command line, but parse_list() doesn't seem to be the right thing here. seems like it should do a arg.split(",") no?
      • 2022-04-14 10431, 2022

      • lucifer
        docker exec -it listenbrainz-spark-reader-prod python3 manage.py spark request_model --rank 100 --rank 200
      • 2022-04-14 10443, 2022

      • mayhem
        ahhhh
      • 2022-04-14 10447, 2022

      • mayhem
        thx
      • 2022-04-14 10449, 2022

      • lucifer
        np
      • 2022-04-14 10438, 2022

      • mayhem
      • 2022-04-14 10408, 2022

      • lucifer
        mayhem: spark-reader-prod container
      • 2022-04-14 10415, 2022

      • lucifer
        cron isn't updated yet
      • 2022-04-14 10445, 2022

      • mayhem
        that was it! thanks!
      • 2022-04-14 10402, 2022

      • lucifer
        2022-04-14 14:11:37,997 listenbrainz_spark.request_consumer.request_consumer INFO Params: {'ranks': [80, 90, 100, 110, 120], 'lambdas': [0.1, 3.0, 6.0, 9.0, 12.0], 'iterations': [10], 'alphas': [3.0]}
      • 2022-04-14 10404, 2022

      • lucifer
        nice
      • 2022-04-14 10412, 2022

      • lucifer
        now lets wait 3 hrs :p
      • 2022-04-14 10429, 2022

      • mayhem
        I doubt it. iterations is set to 10.
      • 2022-04-14 10443, 2022

      • mayhem
        itr 25 made it slow.
      • 2022-04-14 10418, 2022

      • mayhem
        we'll find out soon. :)
      • 2022-04-14 10430, 2022

      • lucifer
        yeah but even itr 10 will take ~5 mins i think, so 25 combinations is 2 hr at least.
      • 2022-04-14 10437, 2022

      • lucifer
        yup makes sense
      • 2022-04-14 10450, 2022

      • mayhem
        well, this is all for testing. for actual production, we'll pick a more constrained range.
      • 2022-04-14 10408, 2022

      • lucifer
        right
      • 2022-04-14 10409, 2022

      • Ansh
        monkey: Yes I agree. I would post a UI mockup this weekend. I’ll also update the proposal to include more details of OAuth.
      • 2022-04-14 10410, 2022

      • Ansh
        Also, I did not get your question.
      • 2022-04-14 10421, 2022

      • Ansh
        How are we delivering the accessToken to the front-end to allow users to submit reviews?
      • 2022-04-14 10411, 2022

      • monkey
        If an access token is stored in the database, then we need a way to send it to the browser when serving queries with authenticated users. What do you have in mind for this?
      • 2022-04-14 10439, 2022

      • Ansh
        Ohh. Got It. I have an idea but have not decided exactly how it would be implemented. I'll work on it and would mention this in the proposal as well.
      • 2022-04-14 10445, 2022

      • mayhem
      • 2022-04-14 10413, 2022

      • lucifer
        ohh nice
      • 2022-04-14 10418, 2022

      • mayhem
        the last runs from today do not include my milligram css changes. are you sure you deployed?
      • 2022-04-14 10415, 2022

      • lucifer
        hmm yeah. let me recheck
      • 2022-04-14 10459, 2022

      • lucifer
        the old ones will still don't have that css but the latest 2 should have had that
      • 2022-04-14 10428, 2022

      • mayhem
        ah, there it is. I suppose I should at the time to the filename so they sort correcter
      • 2022-04-14 10455, 2022

      • lucifer
        ah yes second last one has it
      • 2022-04-14 10433, 2022

      • mayhem
        huh, the error got worse now. good, that represents what I have been seeing.
      • 2022-04-14 10445, 2022

      • mayhem
        things were pretty good when the error was near 6.
      • 2022-04-14 10449, 2022

      • atj
        LB queue is growing
      • 2022-04-14 10450, 2022

      • mayhem
        yep, just looking at that.
      • 2022-04-14 10437, 2022

      • mayhem
        sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) FATAL: password authentication failed for user "listenbrainz_ts"
      • 2022-04-14 10459, 2022

      • mayhem
        does the container need to be rebuilt in order to pick up on the changed password?
      • 2022-04-14 10400, 2022

      • lucifer
        hmm i fixed that yesterday. which container?
      • 2022-04-14 10407, 2022

      • mayhem
        mapping-writer
      • 2022-04-14 10447, 2022

      • lucifer
        yes it needs to be rebased on master and rebuilt
      • 2022-04-14 10458, 2022

      • mayhem
        ok, I can do that.
      • 2022-04-14 10403, 2022

      • lucifer
        i see its on an older version with invalid cache query as well.
      • 2022-04-14 10431, 2022

      • lucifer
        i had done that yestderay, let me see if i can find the image name
      • 2022-04-14 10422, 2022

      • mayhem
        > yes it needs to be rebased on master and rebuilt
      • 2022-04-14 10442, 2022

      • mayhem
        it was running on master, no? just needs rebuiling off master then, yes?
      • 2022-04-14 10449, 2022

      • lucifer
        yes right
      • 2022-04-14 10400, 2022

      • yvanzo
        LB is causing some errors in RabbitMQ: "missed heartbeats from client, timeout: 60s"
      • 2022-04-14 10418, 2022

      • mayhem
        yes, likely part of what we're fixing now.
      • 2022-04-14 10425, 2022

      • atj
        lots of alerts today
      • 2022-04-14 10431, 2022

      • lucifer
        ah no
      • 2022-04-14 10454, 2022

      • lucifer
        that rmq thing is persistent error we get every few seconds.
      • 2022-04-14 10411, 2022

      • yvanzo
        Errors are about both listenbrainz-api-compat-prod and listenbrainz-web-prod
      • 2022-04-14 10415, 2022

      • lucifer
        its a known issue but we don't have a fix yet.
      • 2022-04-14 10427, 2022

      • lucifer
        yup right, we just reconnect and retry currently
      • 2022-04-14 10454, 2022

      • yvanzo
        Ok, that was just fyi as I was looking at logs.
      • 2022-04-14 10455, 2022

      • atj
        do you know the cuase?
      • 2022-04-14 10459, 2022

      • atj
        *cause
      • 2022-04-14 10407, 2022

      • mayhem
      • 2022-04-14 10411, 2022

      • lucifer
        yeah i have a few hunches.
      • 2022-04-14 10415, 2022

      • mayhem
        lucifer: ^^ uncommtted change on gaga
      • 2022-04-14 10424, 2022

      • mayhem
        shit.
      • 2022-04-14 10427, 2022

      • mayhem
        that was smart. :(
      • 2022-04-14 10428, 2022

      • lucifer
        mayhem: you should delete that
      • 2022-04-14 10434, 2022

      • atj
        lol
      • 2022-04-14 10435, 2022

      • lucifer
        yeah :(
      • 2022-04-14 10438, 2022

      • mayhem
        gone.
      • 2022-04-14 10442, 2022

      • lucifer
        thanks!
      • 2022-04-14 10443, 2022

      • mayhem
        but you get the idea.
      • 2022-04-14 10400, 2022

      • lucifer
        yes let me check whether alastairp commited that to docker-server-configs or not
      • 2022-04-14 10438, 2022

      • alastairp
        sorry, I think that might have slipped my mind :(
      • 2022-04-14 10449, 2022

      • lucifer
        np doing that now
      • 2022-04-14 10442, 2022

      • lucifer
        mayhem: https://github.com/metabrainz/docker-server-confi… , try starting that container on map-v-2022-04-13.0
      • 2022-04-14 10447, 2022

      • mayhem
        map-? is normally reserved for the mapping codebase, not the main one. is that intended?
      • 2022-04-14 10414, 2022

      • mayhem
        and I have master build and pushed, so I can just as well use that.
      • 2022-04-14 10435, 2022

      • lucifer
        oh i misunderstood which container
      • 2022-04-14 10444, 2022

      • lucifer
        for that one v-2022-04-13.0 is enough
      • 2022-04-14 10405, 2022

      • mayhem
        I guess then the container wasn't restarted yesterday?
      • 2022-04-14 10434, 2022

      • lucifer
        i remember doing that, not sure what happened.
      • 2022-04-14 10438, 2022

      • lucifer
        f68cf4a1c8cc metabrainz/listenbrainz:v-2022-03-31.0 "/sbin/my_init" 10 minutes ago Up 10 minutes listenbrainz-mbid-mapping-writer-prod
      • 2022-04-14 10411, 2022

      • lucifer
        it shows created 10 min ago? did you restart or it wasn't running all day?
      • 2022-04-14 10420, 2022

      • mayhem
        I restarted.
      • 2022-04-14 10430, 2022

      • mayhem
        now restarted on 4-13.0
      • 2022-04-14 10435, 2022

      • mayhem
        looks good.
      • 2022-04-14 10435, 2022

      • lucifer
        👍
      • 2022-04-14 10453, 2022

      • lucifer
        i think docker-server-configs on gaga was not updated so it started on wrong version
      • 2022-04-14 10400, 2022

      • atj
        some more MB response time spikes happening, this time on patton and ludwig
      • 2022-04-14 10449, 2022

      • atj
        we could do with a better granularity that 5 mins to see how long these actually last
      • 2022-04-14 10455, 2022

      • atj
        s/that/than
      • 2022-04-14 10436, 2022

      • yvanzo
        Is it possible to have UTC time in grafana?
      • 2022-04-14 10448, 2022

      • atj
        user settings I think
      • 2022-04-14 10452, 2022

      • yvanzo
        I’ve set UTC in my preferences but it still displays according to my local time.
      • 2022-04-14 10452, 2022

      • lucifer
        yeah that can be changed from top right
      • 2022-04-14 10422, 2022

      • atj
      • 2022-04-14 10428, 2022

      • atj
        oh, helpful
      • 2022-04-14 10455, 2022

      • mayhem
        and in today's internet shenanigans I am no in possession of a bus ticket from Villahermosa to Guadalajara in Mexico in May. too bad I won't be there then. but I hope whoever thinks their email is mayhem@gmail will sort it out.
      • 2022-04-14 10459, 2022

      • mayhem
        *now
      • 2022-04-14 10417, 2022

      • mayhem
        oh look, a mobile number, lol. let me text this person.
      • 2022-04-14 10448, 2022

      • yvanzo
        lucifer: It applies only to the current dashboard :/
      • 2022-04-14 10416, 2022

      • lucifer
        ah yes :(
      • 2022-04-14 10450, 2022

      • yvanzo
      • 2022-04-14 10407, 2022

      • atj
        if you click on the time period selection drop down you can change it
      • 2022-04-14 10407, 2022

      • yvanzo
        I wonder if it changes just for me or for everyone.
      • 2022-04-14 10409, 2022

      • atj
      • 2022-04-14 10422, 2022

      • lucifer
        local time.
      • 2022-04-14 10425, 2022

      • yvanzo
        atj: Yes, this is what I just did.
      • 2022-04-14 10441, 2022

      • yvanzo
        lucifer: Ok thanks, so it is changing just for the user.
      • 2022-04-14 10442, 2022

      • atj
        why do they have the setting in user prefs
      • 2022-04-14 10449, 2022

      • atj
        and in the time selector
      • 2022-04-14 10457, 2022

      • yvanzo
        It seems that the timezone in profile is ignored.
      • 2022-04-14 10402, 2022

      • atj
        sigh
      • 2022-04-14 10431, 2022

      • lucifer
        atj: so my hunch about the issue rabbitmq issue in LB is that, we have lots of rabbitmq connections pooled together but most of them remain idle for most of the time. unless there is activity heartbeats are missed (well, heartbeats can be sent in the background but that part is either bugged in the library we use or the connection pool). the proposed solution is switch the rmq library and don't pool connections but channels.
      • 2022-04-14 10431, 2022

      • lucifer
        but we don't know if this will work. its all a hunch.
      • 2022-04-14 10432, 2022

      • atj
        ok, so likely application side then
      • 2022-04-14 10434, 2022

      • lucifer
        i have asked multiple times in rmq forums about this but never received a reply.
      • 2022-04-14 10441, 2022

      • lucifer
        yes its an application side issue
      • 2022-04-14 10410, 2022

      • lucifer
        mayhem proposed to blog about it and see if someone shows up then but i want to try out our solution once first.
      • 2022-04-14 10451, 2022

      • mayhem
        cry for help on the blog is a really effective way of getting help.
      • 2022-04-14 10458, 2022

      • lucifer
        i don't disagree but i think we need more visibility into the problem so that someone with knowledge of rmq will be able to help.
      • 2022-04-14 10443, 2022

      • bitmap
        moin!
      • 2022-04-14 10433, 2022

      • mayhem
        moin bitmap
      • 2022-04-14 10423, 2022

      • bitmap
        just a reminder floyd will be restarted in about 45 minutes, I set some banner messages on MB
      • 2022-04-14 10415, 2022

      • BrainzGit
        [listenbrainz-server] 14Fanny-Yuan opened pull request #1954 (03master…dijiasubmit1): [LB-1080] - Remove the bottom whitespace in pages with no BrainzPlayer https://github.com/metabrainz/listenbrainz-server…
      • 2022-04-14 10433, 2022

      • Dijia joined the channel
      • 2022-04-14 10410, 2022

      • MRiddickW joined the channel
      • 2022-04-14 10441, 2022

      • CandyBrain joined the channel
      • 2022-04-14 10415, 2022

      • lucifer
        atj: feel free to edit directly! :)
      • 2022-04-14 10408, 2022

      • BrainzGit
        [listenbrainz-server] 14amCap1712 merged pull request #1953 (03master…log-best-model): Some enhancements to model training https://github.com/metabrainz/listenbrainz-server…
      • 2022-04-14 10408, 2022

      • bitmap
        lucifer: ^ in case you didn't see about floyd & want to stop any LB cron jobs ahead of time
      • 2022-04-14 10415, 2022

      • atj
        lucifer: thanks :)
      • 2022-04-14 10434, 2022

      • lucifer
        bitmap: thanks for reminding. i don't think we have any scheduled for sometime now but i'll shut it down anyway for the time being to be safe.
      • 2022-04-14 10422, 2022

      • bitmap
        👍
      • 2022-04-14 10423, 2022

      • lucifer
        mayhem: shutting down spark reader and cron both, if recs complete they'll get stored in rmq and be written later after downtime.