#metabrainz

/

      • MajorLurker joined the channel
      • 2021-03-02 06153, 2021

      • MajorLurker has quit
      • 2021-03-02 06106, 2021

      • d4rkie has quit
      • 2021-03-02 06146, 2021

      • Nyanko-sensei joined the channel
      • 2021-03-02 06157, 2021

      • SothoTalKer has quit
      • 2021-03-02 06131, 2021

      • SothoTalKer joined the channel
      • 2021-03-02 06152, 2021

      • BrainzGit has quit
      • 2021-03-02 06125, 2021

      • BrainzGit joined the channel
      • 2021-03-02 06114, 2021

      • i7c has quit
      • 2021-03-02 06104, 2021

      • i7c joined the channel
      • 2021-03-02 06104, 2021

      • MajorLurker joined the channel
      • 2021-03-02 06134, 2021

      • MajorLurker has quit
      • 2021-03-02 06143, 2021

      • zas
        yvanzo: sir-prod failed due to consul-template errors
      • 2021-03-02 06147, 2021

      • zas
      • 2021-03-02 06108, 2021

      • zas
        I tried to restart it but now it fails with a Python error
      • 2021-03-02 06109, 2021

      • zas
        I'll stop it for now, please have a look when you can
      • 2021-03-02 06143, 2021

      • zas
        I guess the error is because of the high number of messages it has to process
      • 2021-03-02 06151, 2021

      • zas
        RuntimeError: maximum recursion depth exceeded in cmp
      • 2021-03-02 06147, 2021

      • zas
        there was an issue with hetzner network around 2AM
      • 2021-03-02 06156, 2021

      • zas
        2:00->2:24
      • 2021-03-02 06158, 2021

      • zas
      • 2021-03-02 06127, 2021

      • zas
        but the sir issue started yesterday around 19:00 UTC
      • 2021-03-02 06115, 2021

      • yvanzo
        hi zas: ok, on it.
      • 2021-03-02 06154, 2021

      • zas
        mooooin
      • 2021-03-02 06108, 2021

      • v6lur joined the channel
      • 2021-03-02 06117, 2021

      • v6lur has quit
      • 2021-03-02 06151, 2021

      • MajorLurker joined the channel
      • 2021-03-02 06134, 2021

      • MajorLurker has quit
      • 2021-03-02 06121, 2021

      • BrainzGit
        [musicbrainz-server] reosarevok opened pull request #1950 (master…MBS-11419): MBS-11419: Show YouTube Music links on the sidebar https://github.com/metabrainz/musicbrainz-server/…
      • 2021-03-02 06135, 2021

      • zas
        yvanzo: how did you fix the sir problem?
      • 2021-03-02 06126, 2021

      • yvanzo
        zas: same as last month: I saved all messages to disk and requeued them gradually.
      • 2021-03-02 06150, 2021

      • zas
        cannot sir be fixed so it recovers by itself?
      • 2021-03-02 06154, 2021

      • yvanzo
        done, it's back to normal now.
      • 2021-03-02 06104, 2021

      • yvanzo
        yes, it should be fixed
      • 2021-03-02 06117, 2021

      • v6lur joined the channel
      • 2021-03-02 06135, 2021

      • v6lur has quit
      • 2021-03-02 06147, 2021

      • atj
        I forsee some libertarians suddenly not so keen on free market economics
      • 2021-03-02 06144, 2021

      • reosarevok
        atj: what has sir done to libertarians :D
      • 2021-03-02 06151, 2021

      • atj
        argh
      • 2021-03-02 06136, 2021

      • atj
        my SSH session stalled and I must have pressed up / enter
      • 2021-03-02 06156, 2021

      • atj
        good start to the day!
      • 2021-03-02 06101, 2021

      • repo has quit
      • 2021-03-02 06112, 2021

      • repo joined the channel
      • 2021-03-02 06145, 2021

      • Mr_Monkey
        Good time-of-day !
      • 2021-03-02 06102, 2021

      • ruaok
        tod!
      • 2021-03-02 06119, 2021

      • reosarevok
        bitmap: I'm pretty sure we talked about the each() issue from https://tickets.metabrainz.org/browse/MBS-11409 at some point, but can't remember more now. Do you remember?
      • 2021-03-02 06120, 2021

      • BrainzBot
        MBS-11409: Recurring warning in server logs for recording's page
      • 2021-03-02 06101, 2021

      • Mr_Monkey
        atj: I must say this makes for a great everyday waking-up mantra : "I foresee some libertarians suddenly not so keen on free market economics"
      • 2021-03-02 06151, 2021

      • alastairp
        morning
      • 2021-03-02 06159, 2021

      • BrainzGit
        [musicbrainz-server] reosarevok opened pull request #1951 (master…MBS-9836): MBS-9836: Guess Case: stop uppercasing "the" in artist names https://github.com/metabrainz/musicbrainz-server/…
      • 2021-03-02 06133, 2021

      • reosarevok
        yvanzo: around?
      • 2021-03-02 06145, 2021

      • reosarevok
        I was wondering about why your suggestion in https://tickets.metabrainz.org/browse/MBS-9826 is to start showing hours at 48
      • 2021-03-02 06146, 2021

      • BrainzBot
        MBS-9826: Make edit expiration time show days + hours when it will take more than a day
      • 2021-03-02 06138, 2021

      • atj
        Mr_Monkey: I say it five times in the mirror every morning. Unfortunately I've been unable to summon Rush Limbaugh so far.
      • 2021-03-02 06150, 2021

      • alastairp
        _lucifer: hi, around?
      • 2021-03-02 06108, 2021

      • Mr_Monkey
        Hah !
      • 2021-03-02 06132, 2021

      • BrainzGit
        [critiquebrainz] alastair merged pull request #345 (master…update-translations): Update translations https://github.com/metabrainz/critiquebrainz/pull…
      • 2021-03-02 06135, 2021

      • BrainzGit
        [critiquebrainz] alastair closed pull request #335 (master…dependabot/npm_and_yarn/bootstrap-4.1.2): [Security] Bump bootstrap from 3.4.1 to 4.1.2 https://github.com/metabrainz/critiquebrainz/pull…
      • 2021-03-02 06152, 2021

      • yvanzo
        reosarevok: to bring more accuracy for the next day
      • 2021-03-02 06147, 2021

      • reosarevok
        Do see if my suggestion in the ticket seems good enough maybe :)
      • 2021-03-02 06147, 2021

      • yvanzo
        reosarevok: the WIP commit is supposed to fix the first commit (of PR #1112) or that is more complicated?
      • 2021-03-02 06120, 2021

      • reosarevok
        I think it should just fix the first commit to work with the editor data changes
      • 2021-03-02 06130, 2021

      • reosarevok
        (I wrote this a while ago, but that's what it looked like on review + test)
      • 2021-03-02 06116, 2021

      • zas
        yvanzo: I wonder why solr nodes have that much load when SIR is working, I guess that's related to Indexes replication, but it seems a lot of load for few changes
      • 2021-03-02 06138, 2021

      • zas
      • 2021-03-02 06159, 2021

      • zas
        mb-solr-5 is writing to disk at a permanent rate of 600mb/s ... other nodes < 30mb/s, any idea?
      • 2021-03-02 06123, 2021

      • atj
        the graph shows it's reading from disk at 600MB/s?
      • 2021-03-02 06143, 2021

      • atj
        is the CPU usage user mode usage or non-idle?
      • 2021-03-02 06104, 2021

      • zas
        yes, sorry s/write/read/
      • 2021-03-02 06156, 2021

      • zas
        user
      • 2021-03-02 06119, 2021

      • atj
        some nice graphs there zas, did you create the dashboard?
      • 2021-03-02 06125, 2021

      • zas
        yes
      • 2021-03-02 06141, 2021

      • atj
        I do enjoy creating a Grafana dashboard :)
      • 2021-03-02 06142, 2021

      • zas
        I'll stop this instance, and see how the cluster acts
      • 2021-03-02 06106, 2021

      • zas
        atj: that's a very good monitoring tool, especially since they added alerts
      • 2021-03-02 06143, 2021

      • atj
        yeah it's very useful for operational monitoring. I think the alerting functionality is a bit weak though as you can't use templated graphs.
      • 2021-03-02 06111, 2021

      • atj
        quite strange how the disk read on solr-5 seems to be the only outlier
      • 2021-03-02 06124, 2021

      • zas
        yes... I'd more than strange ;)
      • 2021-03-02 06106, 2021

      • zas
        ok stopped this solr instance, let's see if other nodes exhibits same behavior
      • 2021-03-02 06118, 2021

      • zas
        it doesn't seem another node started to read at this rate, in fact, apart a bit more handled queries (4 nodes instead of 5) nothing changed
      • 2021-03-02 06124, 2021

      • zas
        I'll start it again
      • 2021-03-02 06149, 2021

      • zas
        rebooting the node while at it
      • 2021-03-02 06113, 2021

      • alastairp
        deploying new LB
      • 2021-03-02 06133, 2021

      • Etua joined the channel
      • 2021-03-02 06110, 2021

      • BrainzGit
        [listenbrainz-server] paramsingh opened pull request #1309 (master…param-user-recommendation-event-table): Add new table for user recommendation events https://github.com/metabrainz/listenbrainz-server…
      • 2021-03-02 06126, 2021

      • zas
        atj: the template variable support for grafana's alerts is in the work: https://github.com/grafana/grafana/issues/6557#is…
      • 2021-03-02 06110, 2021

      • zas
        ok, the insane read rate issue disappeared after solr restart...
      • 2021-03-02 06114, 2021

      • _lucifer
        alastairp: hi! i am available now
      • 2021-03-02 06125, 2021

      • alastairp
        _lucifer: hi, I just restarted cron
      • 2021-03-02 06146, 2021

      • Etua has quit
      • 2021-03-02 06148, 2021

      • alastairp
        about to do timescale-reader, spotify-writer, and spark
      • 2021-03-02 06126, 2021

      • _lucifer
        cool, i have logged in to lemmy as well
      • 2021-03-02 06134, 2021

      • atj
        zas: thanks, I remember looking at that ticket some time ago and seeing 100 odd "+1" comments!
      • 2021-03-02 06109, 2021

      • _lucifer
        alastairp: i was wondering why the registrator did not restart in case of lb containers like it happened in case of CB
      • 2021-03-02 06124, 2021

      • alastairp
        _lucifer: because I'm starting services manually
      • 2021-03-02 06132, 2021

      • _lucifer
        makes sense
      • 2021-03-02 06143, 2021

      • alastairp
        `source scripts/services.sh` and then manually run `start_listenbrainz_api_compat 'prod' 'v-2021-03-01.1'`
      • 2021-03-02 06125, 2021

      • _lucifer
        in the meanwhile, should i build beta and test images so that we can update them as well in everything goes right in prod
      • 2021-03-02 06138, 2021

      • alastairp
        sure, go ahead with building beta
      • 2021-03-02 06150, 2021

      • alastairp
        test isn't too important - we only use that for testing major changes
      • 2021-03-02 06106, 2021

      • _lucifer
        👍
      • 2021-03-02 06146, 2021

      • alastairp
        restarting web
      • 2021-03-02 06105, 2021

      • ruaok
        do labs-api soon, plz. the new web relies on labs-api.
      • 2021-03-02 06106, 2021

      • alastairp
        LB back up
      • 2021-03-02 06111, 2021

      • alastairp
        yes, I did both at the same time
      • 2021-03-02 06121, 2021

      • ruaok
        👍
      • 2021-03-02 06126, 2021

      • alastairp
        all containers now running 2021-03-01.1
      • 2021-03-02 06135, 2021

      • _lucifer
        !m alastairp
      • 2021-03-02 06135, 2021

      • BrainzBot
        You're doing good work, alastairp!
      • 2021-03-02 06141, 2021

      • alastairp
        quiet, nagios
      • 2021-03-02 06108, 2021

      • ruaok
        crap, the labs api throws errors. I'm on it.
      • 2021-03-02 06148, 2021

      • alastairp
        I see it. misconfigured typesense url?
      • 2021-03-02 06113, 2021

      • ruaok
        likely. in consul temp?
      • 2021-03-02 06131, 2021

      • _lucifer
        the api key seems to be wrong
      • 2021-03-02 06107, 2021

      • ruaok
        it throws a 404. would it do that for a wrong api key?
      • 2021-03-02 06128, 2021

      • alastairp
        ruaok: do you want me to roll back to old web and labs-api for now, or do you think that we can fix it shortly?
      • 2021-03-02 06140, 2021

      • ruaok
        I can fix shortly.
      • 2021-03-02 06130, 2021

      • _lucifer
        fair point!
      • 2021-03-02 06138, 2021

      • _lucifer
        alastairp: beta's pushed
      • 2021-03-02 06150, 2021

      • ruaok
        its a cannot connect problem. so URL construction goes wrong somehow.
      • 2021-03-02 06128, 2021

      • alastairp
        ruaok: is it possible that the mapping isn't there?
      • 2021-03-02 06143, 2021

      • alastairp
        curl http://10.2.2.31:8108/collections/ -H "Content-Type: application/json" -H "X-TYPESENSE-API-KEY: ---the---key---"
      • 2021-03-02 06148, 2021

      • alastairp
        [{"created_at":1614658498,"default_sorting_field":"score","fields":[{"facet":false,"name":"combined","optional":false,"type":"string"},{"facet":false,"name":"score","optional":false,"type":"int32"}],"name":"mbid_mapping_20210302_041458","num_documents":15283426,"num_memory_shards":4}]
      • 2021-03-02 06159, 2021

      • alastairp
        seems that it didn't rename it from mbid_mapping_20210302_041458 to mbid_mapping
      • 2021-03-02 06136, 2021

      • ruaok
        there is a concept of collection aliases. there is an alias from mbid_mapping to mbid_mapping_20210302_041458. Changing this alias allows the atomic rotating of indexes.
      • 2021-03-02 06112, 2021

      • ruaok
        this problem is a connection problem first and foremost. the config info that LB labs has is correct, but the container cannot reach typesense.
      • 2021-03-02 06147, 2021

      • antlarr2 is now known as antlarr
      • 2021-03-02 06150, 2021

      • alastairp
        the alias is called mbid_mapping_latest, but the config at https://github.com/metabrainz/listenbrainz-server… is looking for mbid_mapping
      • 2021-03-02 06111, 2021

      • ruaok
        uhm, for some reason the connection now works.
      • 2021-03-02 06114, 2021

      • ruaok
        huh.
      • 2021-03-02 06100, 2021

      • ruaok
        ok, agreed, that makes sense.
      • 2021-03-02 06121, 2021

      • ruaok
        I'll just commit the fix directly to master. should I push v-2021-03-02.1 ?
      • 2021-03-02 06151, 2021

      • alastairp
        just do the commit, _lucifer can do the release + push :)
      • 2021-03-02 06159, 2021

      • ruaok
        commit pushed.
      • 2021-03-02 06101, 2021

      • ruaok
        thanks _lucifer
      • 2021-03-02 06101, 2021

      • alastairp
        it'll be 02.0, I released yesterday's build
      • 2021-03-02 06119, 2021

      • _lucifer
        on it
      • 2021-03-02 06151, 2021

      • alastairp
        _lucifer: git tag v-2021-03-02.1, merge master to production, make github release, build+push image
      • 2021-03-02 06107, 2021

      • alastairp
        uh
      • 2021-03-02 06109, 2021

      • alastairp
        v-2021-03-02.0
      • 2021-03-02 06140, 2021

      • alastairp
        I'm really not a fan of magic python HTTP API wrappers that hide the details of the request that it's making. all such clients should have a straightforward way of debugging the details of the requests
      • 2021-03-02 06114, 2021

      • ruaok
        "magic python HTTP API wrappers" like what?
      • 2021-03-02 06125, 2021

      • alastairp
        like the typesense client library
      • 2021-03-02 06153, 2021

      • alastairp
        I call client.collections['foo'].documents.search, but what http request is that actually making
      • 2021-03-02 06124, 2021

      • alastairp
        I ran into exactly the same problem with another client yesterday. Once I could work out what query it was trying to make, the problem was obvious, but it took ages to get to that point
      • 2021-03-02 06102, 2021

      • ruaok
        I HATE that API bit.
      • 2021-03-02 06119, 2021

      • ruaok
        terrible for testing as well.
      • 2021-03-02 06138, 2021

      • BrainzGit
        [listenbrainz-server] release v-2021-03-02.0 has been published by github-actions[bot]: https://github.com/metabrainz/listenbrainz-server…
      • 2021-03-02 06106, 2021

      • alastairp
        OK. the only remaining python apps on the old version of consul-template are sir and wikidata bot
      • 2021-03-02 06108, 2021

      • alastairp
        !m _lucifer