#metabrainz

/

      • ruaok
        and judging by the looks of the recents page, we're now 38 minutes behind.
      • 2020-02-12 04347, 2020

      • ruaok
        that is, if spotify is not behind too.
      • 2020-02-12 04312, 2020

      • ruaok
        it seems we store lastest_listened at, but we do not use it. at least I can't see where it gets used. lol.
      • 2020-02-12 04300, 2020

      • iliekcomputers
      • 2020-02-12 04322, 2020

      • iliekcomputers
        It's used to get the order in which we make requests to the Spotify API
      • 2020-02-12 04329, 2020

      • ruaok
        ahhh, I see.
      • 2020-02-12 04342, 2020

      • ruaok
        explains why I found no references in the code itself.
      • 2020-02-12 04329, 2020

      • iliekcomputers
        That's where https://github.com/livegrep/livegrep would have been really helpful. :D cc zas
      • 2020-02-12 04326, 2020

      • ruaok
        zas is already working on it.
      • 2020-02-12 04358, 2020

      • Etua joined the channel
      • 2020-02-12 04331, 2020

      • BrainzGit
        [listenbrainz-server] mayhem opened pull request #727 (master…no-spotify-dups): Do not submit previously submitted listens to the pipeline. untested! https://github.com/metabrainz/listenbrainz-server…
      • 2020-02-12 04359, 2020

      • ruaok
        totally untested. I need to run off to lunch, but if you get a moment, sanity check my approach?
      • 2020-02-12 04303, 2020

      • ruaok zooms off
      • 2020-02-12 04347, 2020

      • iliekcomputers
        ruaok: looks reasonable to me.
      • 2020-02-12 04309, 2020

      • iliekcomputers
        We could probably avoid the extra database call from getting the user again
      • 2020-02-12 04307, 2020

      • Sophist_UK has quit
      • 2020-02-12 04303, 2020

      • Gazooo has quit
      • 2020-02-12 04347, 2020

      • Gazooo joined the channel
      • 2020-02-12 04327, 2020

      • eharris has quit
      • 2020-02-12 04305, 2020

      • Etua has quit
      • 2020-02-12 04349, 2020

      • Sophist-UK joined the channel
      • 2020-02-12 04301, 2020

      • Sophist-UK has quit
      • 2020-02-12 04358, 2020

      • Sophist-UK joined the channel
      • 2020-02-12 04334, 2020

      • Lotheric has quit
      • 2020-02-12 04312, 2020

      • shivam-kapila has quit
      • 2020-02-12 04311, 2020

      • eharris joined the channel
      • 2020-02-12 04352, 2020

      • ruaok
        hmm. ok, I can look again and see if that is doable.
      • 2020-02-12 04340, 2020

      • endurance21 joined the channel
      • 2020-02-12 04321, 2020

      • sarthak_jain joined the channel
      • 2020-02-12 04302, 2020

      • sampsyo has quit
      • 2020-02-12 04350, 2020

      • sarthak_jain
        Hey pristine__
      • 2020-02-12 04337, 2020

      • sarthak_jain
        When I run `./spark_test.sh` locally, I am facing error stating `cannot create temporary directory` ?
      • 2020-02-12 04342, 2020

      • sarthak_jain
        Any idea?
      • 2020-02-12 04347, 2020

      • sarthak_jain
        It was working fine earlier
      • 2020-02-12 04358, 2020

      • pristine__
        Paate the complete log please
      • 2020-02-12 04344, 2020

      • sarthak_jain
        Here you go !
      • 2020-02-12 04345, 2020

      • sarthak_jain
      • 2020-02-12 04317, 2020

      • endurance21 has quit
      • 2020-02-12 04340, 2020

      • endurance21 joined the channel
      • 2020-02-12 04306, 2020

      • endurance21 has quit
      • 2020-02-12 04344, 2020

      • ruaok
        iliekcomputers: I dont quite see an easy way to avoid the extra user fetch -- from what I can see we'd need to cache them in ram someplace and then ensure we're cleaning up, etc.
      • 2020-02-12 04352, 2020

      • ruaok
        am I missing an easy approach that you see?
      • 2020-02-12 04355, 2020

      • iliekcomputers
        ruaok: the `user` object passed to `process_one_user` is an instance of this class: https://github.com/metabrainz/listenbrainz-server…
      • 2020-02-12 04310, 2020

      • iliekcomputers
        So it should already have the latest_listened_at property
      • 2020-02-12 04315, 2020

      • ruaok
        oh! right. I first thought that was a lb user object, not spotify, then realized that that was not the case.
      • 2020-02-12 04317, 2020

      • ruaok
        lemme fix.
      • 2020-02-12 04321, 2020

      • ruaok
        pushed.
      • 2020-02-12 04327, 2020

      • iliekcomputers
        looks good to me, do you have a local environment to test?
      • 2020-02-12 04302, 2020

      • ruaok
        not really.
      • 2020-02-12 04328, 2020

      • iliekcomputers
        yolo it and trust the automated tests?
      • 2020-02-12 04348, 2020

      • ruaok
        my plan was to add some debug statements and ensure that the right thing is happening and carefully watch for a while.
      • 2020-02-12 04309, 2020

      • ruaok
        thankfully this is slightly less mission critical than the web site staying up.
      • 2020-02-12 04331, 2020

      • ruaok
        so, not quite 'yolo it', but close. more like good ole testing in production. :)
      • 2020-02-12 04351, 2020

      • ruaok
        but, given that I want to watch it over time, I'll wait until tomorrow to test it.
      • 2020-02-12 04358, 2020

      • iliekcomputers
        sounds good
      • 2020-02-12 04311, 2020

      • ruaok
        I'm still not 100% convinced that this is the root cause, but it should at least alleviate some symptoms.
      • 2020-02-12 04323, 2020

      • sarthak_jain has quit
      • 2020-02-12 04342, 2020

      • zas
        experimental setup for livegrep is available at http://livegrep.metabrainz.org:8910 (WIP), please test and report issues, repos are re-indexed every 5 mins
      • 2020-02-12 04354, 2020

      • ruaok
        ohh, I like it.
      • 2020-02-12 04324, 2020

      • ruaok
        I wanted to show pristine__ a badly named function and I couldn't find it. I was looking in the wrong codebase.
      • 2020-02-12 04311, 2020

      • zas
        it indexes only public repos
      • 2020-02-12 04322, 2020

      • zas
        but all metabrainz repos
      • 2020-02-12 04352, 2020

      • alastairp
        it's _fast_
      • 2020-02-12 04353, 2020

      • alastairp
      • 2020-02-12 04358, 2020

      • ruaok
      • 2020-02-12 04313, 2020

      • zas
        I'll prolly add a nginx instance in front of it
      • 2020-02-12 04314, 2020

      • ruaok
        that is a badly named function and I'm guilty for it. 🤣
      • 2020-02-12 04334, 2020

      • alastairp
        hey, you asked for it. I gave it to you
      • 2020-02-12 04347, 2020

      • ruaok
        no complaints.
      • 2020-02-12 04351, 2020

      • pristine__
        :p
      • 2020-02-12 04352, 2020

      • alastairp
        not from me either
      • 2020-02-12 04321, 2020

      • ruaok
        ok, I quite like livegrep. thanks zas, iliekcomputers
      • 2020-02-12 04357, 2020

      • iliekcomputers
        thank you zas!
      • 2020-02-12 04338, 2020

      • ruaok
        yeah, I'm sold. :)
      • 2020-02-12 04305, 2020

      • zas
        np, keep in mind that's not a definitive install, more work is needed for "production" but it works (tm)
      • 2020-02-12 04321, 2020

      • ruaok
        iliekcomputers: we are getting more traffic for LB -- more people signing up for saving their listens. so that is promising!
      • 2020-02-12 04330, 2020

      • zas
        and it runs fast on the smallest hetzner VM :)
      • 2020-02-12 04303, 2020

      • iliekcomputers
        ruaok: are you saying that because of the queues?
      • 2020-02-12 04316, 2020

      • ruaok
        more because of the recents page: https://listenbrainz.org/recent
      • 2020-02-12 04343, 2020

      • ruaok
        a greater variety of users and less time between individual listens.
      • 2020-02-12 04311, 2020

      • iliekcomputers
        ah
      • 2020-02-12 04341, 2020

      • iliekcomputers
        could we have a graph for the number of post requests to /1/submit_listens
      • 2020-02-12 04351, 2020

      • iliekcomputers
        (or do we already)
      • 2020-02-12 04305, 2020

      • ruaok
        mb_stats is also running for the LB API endpoint, so we get some insight.
      • 2020-02-12 04315, 2020

      • ruaok
        but, not a graph.
      • 2020-02-12 04324, 2020

      • ruaok
        zas: can you do that with grafana?
      • 2020-02-12 04355, 2020

      • iliekcomputers
        link for mb_stats, sorry I missed it
      • 2020-02-12 04306, 2020

      • zas
      • 2020-02-12 04327, 2020

      • apetresc has quit
      • 2020-02-12 04345, 2020

      • alastairp
        editing question: is there are policy for how to use work aliases for transliterations?
      • 2020-02-12 04311, 2020

      • alastairp
        (or perhaps more specifically, is that the correct way of doing it? I don't see a work-work transliteration rel type. perhaps you could do rel-rel and create a new work and make the same recording-work relation as on the original release?)
      • 2020-02-12 04335, 2020

      • reosarevok
        Why would you have a new work for a transliteration? That's just a name change, no?
      • 2020-02-12 04307, 2020

      • reosarevok
        Work aliases seem like the only reasonable way, we don't have script for those at the moment but it's better than nothing
      • 2020-02-12 04328, 2020

      • ruaok
      • 2020-02-12 04359, 2020

      • ruaok
        passwd in syswiki:TopStuff.md
      • 2020-02-12 04309, 2020

      • ruaok
        that is the top api page I was talking about.
      • 2020-02-12 04300, 2020

      • alastairp
        ah, script! that's the field we would want
      • 2020-02-12 04308, 2020

      • sampsyo joined the channel
      • 2020-02-12 04313, 2020

      • ruaok
        however, it is really interesting how the response time degrades when the queue is backed up.
      • 2020-02-12 04321, 2020

      • ruaok
        which makes no sense to me yet.
      • 2020-02-12 04334, 2020

      • alastairp
        no strong reason to have a new work for a transliteration. alias feels better to me too. script would be great, but for now not a problem if we don't havve it. thanks
      • 2020-02-12 04310, 2020

      • apetresc joined the channel
      • 2020-02-12 04320, 2020

      • iliekcomputers
        ruaok: response times of the submit listens endpoint or response times in general?
      • 2020-02-12 04336, 2020

      • iliekcomputers
        It might be because of influx gets overloaded.
      • 2020-02-12 04315, 2020

      • reosarevok
        bitmap, yvanzo: around today?
      • 2020-02-12 04321, 2020

      • chaban
        Is the FixTrackLength script still running regularly or are these two bugs? https://tickets.metabrainz.org/browse/MBS-10602
      • 2020-02-12 04322, 2020

      • BrainzBot
        MBS-10602: Track lengths removed with late applied "edit medium" edit despite existing DiscID
      • 2020-02-12 04354, 2020

      • reosarevok
        It might be that it hasn't been readded automatically because it can see it was "hand-set to empty"?
      • 2020-02-12 04327, 2020

      • chaban
        That would be stupid.
      • 2020-02-12 04351, 2020

      • Sophist-UK has quit
      • 2020-02-12 04311, 2020

      • reosarevok
        I mean, if all the code checks is "this hasn't been touched by an edit", it'd make sense
      • 2020-02-12 04319, 2020

      • reosarevok
        I haven't checked the code, just a guess
      • 2020-02-12 04344, 2020

      • bitmap
        reosarevok: I'll be here in a bit
      • 2020-02-12 04301, 2020

      • yvanzo
        same
      • 2020-02-12 04346, 2020

      • Sophist-UK joined the channel
      • 2020-02-12 04342, 2020

      • yvanzo
        pouik
      • 2020-02-12 04336, 2020

      • eharris has quit
      • 2020-02-12 04327, 2020

      • moufl has quit
      • 2020-02-12 04325, 2020

      • eharris joined the channel
      • 2020-02-12 04346, 2020

      • ruaok
        iliekcomputers: response times of the API... and yes, it could be an influx issue.
      • 2020-02-12 04309, 2020

      • ruaok
      • 2020-02-12 04347, 2020

      • ruaok
        actually, the number of requests to the API jumped dramatically, which might be saturating things, causing wait times for new back-ends....
      • 2020-02-12 04342, 2020

      • CatQuest
      • 2020-02-12 04332, 2020

      • reosarevok
        Probably once it's out of beta :/
      • 2020-02-12 04337, 2020

      • reosarevok
        (useful!)
      • 2020-02-12 04358, 2020

      • CatQuest
        eh how can i test it then? :P
      • 2020-02-12 04324, 2020

      • CatQuest
        also reo ಠ_ಠ
      • 2020-02-12 04355, 2020

      • BrainzGit
        [mb-solr] dependabot[bot] opened pull request #35 (master…dependabot/maven/mb-solr/org.apache.solr-solr-core-8.4.0): Bump solr-core from 7.7.2 to 8.4.0 in /mb-solr https://github.com/metabrainz/mb-solr/pull/35
      • 2020-02-12 04309, 2020

      • reosarevok
        You might not be able too, hence "useful!"
      • 2020-02-12 04311, 2020

      • reosarevok
        *to
      • 2020-02-12 04322, 2020

      • reosarevok
        bitmap: how bit is your bit :D
      • 2020-02-12 04310, 2020

      • yvanzo
        We can run it on test.mb.o if you want to.
      • 2020-02-12 04321, 2020

      • reosarevok
        How old is that data?
      • 2020-02-12 04335, 2020

      • CatQuest
        mega old
      • 2020-02-12 04343, 2020

      • CatQuest
        test shoudl be updated
      • 2020-02-12 04333, 2020

      • yvanzo
        where does that dependabot PR comes from?
      • 2020-02-12 04339, 2020

      • reosarevok
        Magic
      • 2020-02-12 04346, 2020

      • CatQuest
        oh no.
      • 2020-02-12 04331, 2020

      • CatQuest
        but. see Clarke's law
      • 2020-02-12 04318, 2020

      • BrainzGit
        [mb-solr] yvanzo closed pull request #35 (master…dependabot/maven/mb-solr/org.apache.solr-solr-core-8.4.0): Bump solr-core from 7.7.2 to 8.4.0 in /mb-solr https://github.com/metabrainz/mb-solr/pull/35
      • 2020-02-12 04321, 2020

      • yvanzo
        apprentice’s magic
      • 2020-02-12 04319, 2020

      • prabal has quit
      • 2020-02-12 04334, 2020

      • bitmap
        I'm back
      • 2020-02-12 04353, 2020

      • CatQuest
        hi bitmap !
      • 2020-02-12 04319, 2020

      • bitmap
        hi :)
      • 2020-02-12 04332, 2020

      • shivam-kapila joined the channel
      • 2020-02-12 04302, 2020

      • reosarevok
        Hi!
      • 2020-02-12 04340, 2020

      • CatQuest
        REO
      • 2020-02-12 04353, 2020

      • reosarevok
        ⏩ 🚋:
      • 2020-02-12 04316, 2020

      • reosarevok
        legoktm: how does <ref> work in WP? Is that a specific addon or something?