#metabrainz

/

      • nelgin has quit
      • 2020-10-23 29737, 2020

      • sumedh joined the channel
      • 2020-10-23 29722, 2020

      • reosarevok
        bitmap: remind me, how hard would it be to make the change to the work types and whatnot in the pots so that they have the type name as the ID?
      • 2020-10-23 29741, 2020

      • reosarevok
        (I still have a ton of tabs opened to change type names but should probably wait until that is done)
      • 2020-10-23 29713, 2020

      • sumedh has quit
      • 2020-10-23 29736, 2020

      • sumedh joined the channel
      • 2020-10-23 29748, 2020

      • v6lur joined the channel
      • 2020-10-23 29728, 2020

      • adhawkins
        Morning all. Just thought I'd report back about my dockerised MBserver setup. I was getting regular OOM errors from it. Since increasing the RAM to 6 GB nearly 16 days ago I haven't had a single one.
      • 2020-10-23 29721, 2020

      • adhawkins
        Seems a lot, considering the previous VM that ran it worked fine with just 4 Gig. I guess if the docker-compose file can be arranged so as to optionally not start the search indexing stuff that might help?
      • 2020-10-23 29702, 2020

      • Gazooo79494 has quit
      • 2020-10-23 29741, 2020

      • Gazooo79494 joined the channel
      • 2020-10-23 29706, 2020

      • ruaok
        mooooin!
      • 2020-10-23 29726, 2020

      • ruaok
        pristine___: on #1153, that is mostly moving code around, not so much making functional changes, right?
      • 2020-10-23 29735, 2020

      • ruaok wonders how closely he should ready this PR
      • 2020-10-23 29752, 2020

      • ruaok
        but I am glad that you made a separate PR for the refactoring work. 👍
      • 2020-10-23 29738, 2020

      • pristine___
        ruaok: yeah, have just dumped dataframe function in a utils so that create_df_recording and create_df_artist can use it from there
      • 2020-10-23 29757, 2020

      • ruaok
        ok, then it should be fine. let me have one more look.
      • 2020-10-23 29705, 2020

      • ruaok
        great first step!
      • 2020-10-23 29707, 2020

      • ruaok
        should I merge it? (it looks good to me)
      • 2020-10-23 29703, 2020

      • pristine___
        Yeah, sure
      • 2020-10-23 29717, 2020

      • pristine___
        Thanks!
      • 2020-10-23 29735, 2020

      • BrainzGit
        [listenbrainz-server] mayhem merged pull request #1153 (master…dataframe-utils): Utils for shareable dataframe functions https://github.com/metabrainz/listenbrainz-server…
      • 2020-10-23 29702, 2020

      • pristine___
        I think there be a few more (1 or 2) refactor PRs before I start with create_df_artist.
      • 2020-10-23 29721, 2020

      • pristine___
        Then we are good to go!
      • 2020-10-23 29754, 2020

      • ruaok
        seems about right. I'm excited for the refactorings because that will make more models possible.
      • 2020-10-23 29701, 2020

      • ruaok
        user-user would be pretty sexy.
      • 2020-10-23 29709, 2020

      • ruaok
        and I'm dreaming up all sorts of things that could be good models too. I'm also stoked that creatings recs is now fast. ⛄
      • 2020-10-23 29711, 2020

      • ruaok
        alastairp: do you know how to discover and run unit tests from inside a python script?
      • 2020-10-23 29728, 2020

      • alastairp
        hi
      • 2020-10-23 29736, 2020

      • alastairp
        what's your final goal?
      • 2020-10-23 29702, 2020

      • ruaok
        test.sh has `py.test`in it, but given how we're setting up running troi via model invocation, I'd like to add that as one of the top level commands in cli.py
      • 2020-10-23 29715, 2020

      • ruaok
        python -m troi.cli test
      • 2020-10-23 29735, 2020

      • ruaok
        to run tests and get rid of test.sh
      • 2020-10-23 29756, 2020

      • alastairp
      • 2020-10-23 29718, 2020

      • ruaok tries it
      • 2020-10-23 29732, 2020

      • ruaok
        promising. I was trying it via unittest, but I now see that that was the wrong level.
      • 2020-10-23 29757, 2020

      • alastairp
        right, unittest has test discovery stuff that you can do, but I think that keeping it at pytest is good
      • 2020-10-23 29709, 2020

      • ruaok
        pytest.main(["-x", "."])
      • 2020-10-23 29712, 2020

      • ruaok
        does the trick
      • 2020-10-23 29719, 2020

      • alastairp
        there are some interesting ways of writing tests in pytest, which I like
      • 2020-10-23 29742, 2020

      • alastairp
        keep in mind that it'd be good to be able to pass args to pytest - specifically I use -v often, and also passing a single test to it
      • 2020-10-23 29759, 2020

      • ruaok
        I was expecting that. :) on it.
      • 2020-10-23 29755, 2020

      • alastairp
        just related to your PR, do you prefer travis, or you just did it because it was easy? Want me to set it up on jenkins, or...?
      • 2020-10-23 29722, 2020

      • alastairp
        it seems like almost all of our checks are on jenkins atm
      • 2020-10-23 29728, 2020

      • ruaok
        honestly, I am confused by our setup. why do we have two? is one preferred over the other?
      • 2020-10-23 29735, 2020

      • _lucifer
        do docker's new policies affect meb?
      • 2020-10-23 29742, 2020

      • alastairp
        _lucifer: yes
      • 2020-10-23 29747, 2020

      • ruaok
        _lucifer: what are they?
      • 2020-10-23 29749, 2020

      • alastairp
        I have similar questions
      • 2020-10-23 29704, 2020

      • alastairp
        I guess we've had jenkins for a long time,
      • 2020-10-23 29705, 2020

      • _lucifer
        they are limiting number of pulls and image retention period for free accounts
      • 2020-10-23 29706, 2020

      • ruaok
        similar questions about our test setup or docker?
      • 2020-10-23 29712, 2020

      • alastairp
        about test setup
      • 2020-10-23 29702, 2020

      • alastairp
        I don't specifically know why some tests on some projects use travis. I believe that jenkins existed at the time that they were added, but I can't be sure
      • 2020-10-23 29703, 2020

      • ruaok
        zas: ^^ see comment about docker changes. can you please examine if this will impact us and what we might be able to do to mitigate it?
      • 2020-10-23 29714, 2020

      • _lucifer
      • 2020-10-23 29717, 2020

      • alastairp
        I believe bitmap moved some MB tests from travis to jenkins recently
      • 2020-10-23 29733, 2020

      • ruaok
        alastairp: well, if you feel like setting another CI, I don't much care. I'll get the test running reliably and well from the command line in the scope of this PR
      • 2020-10-23 29718, 2020

      • alastairp
        I don't care strongly either, so I guess we're at an impasse :)
      • 2020-10-23 29722, 2020

      • alastairp
        we can leave it, that's fine
      • 2020-10-23 29733, 2020

      • ruaok
        "Free plan – anonymous users: 100 pulls per 6 hours "
      • 2020-10-23 29746, 2020

      • ruaok
        lets leave it until we have a clear indication as to why/what we should do.
      • 2020-10-23 29712, 2020

      • ruaok
        _lucifer: given that we user our own images internally and that others dont pull them, this may not affect us.
      • 2020-10-23 29727, 2020

      • kepstin has quit
      • 2020-10-23 29727, 2020

      • tripathyprateek[ has quit
      • 2020-10-23 29728, 2020

      • goldenshimmer has quit
      • 2020-10-23 29729, 2020

      • MagnusSvensson[m has quit
      • 2020-10-23 29734, 2020

      • dotmavriq[m] has quit
      • 2020-10-23 29739, 2020

      • maxlath[m] has quit
      • 2020-10-23 29746, 2020

      • reg[m] has quit
      • 2020-10-23 29747, 2020

      • joshuaboniface has quit
      • 2020-10-23 29736, 2020

      • _lucifer
        👍
      • 2020-10-23 29745, 2020

      • ruaok
        alastairp: I'll just keep pushing new tests to the fix-basic-tests branch. tell me when you're close to reviewing it so I don't push more stuff in your way.
      • 2020-10-23 29739, 2020

      • alastairp
        ok. I'm not sure how much time I have today (on another project today), but if you think it's small I can probably have a quick look later after lunch
      • 2020-10-23 29701, 2020

      • ruaok
      • 2020-10-23 29708, 2020

      • ruaok
        like that, right?
      • 2020-10-23 29717, 2020

      • ruaok
        alastairp: the PR is still small. maybe I should just stop adding to it in hopes that you can look after lunch.
      • 2020-10-23 29719, 2020

      • ruaok
        I'll do that.
      • 2020-10-23 29708, 2020

      • dotmavriq[m] joined the channel
      • 2020-10-23 29759, 2020

      • alastairp
        perfect to both those points, thaks
      • 2020-10-23 29757, 2020

      • alastairp
        actually, I lied. I've got some time booked with zas at 2 to do some granfa stuff, so I do have time to do some reviews :)
      • 2020-10-23 29722, 2020

      • alastairp
        ruaok: fyi, will deploy a branch to LB beta at 2-ish for an hour or so for testing
      • 2020-10-23 29746, 2020

      • ruaok
        I'll have the branch ready for review before 2 then.
      • 2020-10-23 29746, 2020

      • ruaok
        k
      • 2020-10-23 29712, 2020

      • d4rkie joined the channel
      • 2020-10-23 29707, 2020

      • D4RK-PH0ENiX has quit
      • 2020-10-23 29755, 2020

      • ruaok
        alastairp: https://github.com/metabrainz/troi-recommendation… is ready. plenty long as it is.
      • 2020-10-23 29724, 2020

      • pristine___
        ruaok: you too artist recs this week look similar to last week and a week before. Is that?
      • 2020-10-23 29728, 2020

      • pristine___
        Top*
      • 2020-10-23 29748, 2020

      • ruaok
        I havent been paying attention to top artists as much -- similar gives better results for daily jams, so not sure.
      • 2020-10-23 29702, 2020

      • ruaok
        similar artists are vastly more diverse this week than last.
      • 2020-10-23 29726, 2020

      • ruaok
        I sort of suspect that was related to ishaanshah loading more listens in a giant batch. that probably threw things off.
      • 2020-10-23 29713, 2020

      • pristine___
        Yeah, top artist are not diverse because let's say a user listens to 10 artist last week, so tracks of only 10 artists in recs, but tracks of 10 * 10 artists in similar tracks
      • 2020-10-23 29730, 2020

      • pristine___
        ruaok: are dumps back on track?
      • 2020-10-23 29756, 2020

      • ruaok
        the PR to fix them was just merged, so we should see regular imports very soon.
      • 2020-10-23 29717, 2020

      • ruaok
        hmm. daily-jams for top artists only gives 5 tracks.
      • 2020-10-23 29718, 2020

      • ishaanshah
        pristine___: yes they are
      • 2020-10-23 29710, 2020

      • ruaok
        yeah, my top artists is so massive attack heavy, that the artist limiter tosses out most of the tracks.
      • 2020-10-23 29750, 2020

      • ruaok
        perhaps you are right and need to include some artist limiting at the CF level.
      • 2020-10-23 29701, 2020

      • pristine___
        ishaanshah: can you tell me what is the latest ts of listens in spark cluster? I mean is there a to know that?
      • 2020-10-23 29712, 2020

      • reg[m] joined the channel
      • 2020-10-23 29712, 2020

      • joshuaboniface joined the channel
      • 2020-10-23 29712, 2020

      • maxlath[m] joined the channel
      • 2020-10-23 29712, 2020

      • kepstin joined the channel
      • 2020-10-23 29713, 2020

      • goldenshimmer joined the channel
      • 2020-10-23 29713, 2020

      • tripathyprateek[ joined the channel
      • 2020-10-23 29713, 2020

      • MagnusSvensson[m joined the channel
      • 2020-10-23 29743, 2020

      • pristine___
        ruaok: I would really like to improve top/similar on spark side once I am done with feedback stuff
      • 2020-10-23 29745, 2020

      • ishaanshah
        pristine___: you will have to download latest dump manually and check
      • 2020-10-23 29704, 2020

      • pristine___
        Phew, rough idea?
      • 2020-10-23 29706, 2020

      • ishaanshah
        Incremental dump
      • 2020-10-23 29738, 2020

      • ishaanshah
        Most prolly yesterday 6-8pm UTC
      • 2020-10-23 29749, 2020

      • pristine___
        thanks!
      • 2020-10-23 29726, 2020

      • iliekcomputers
        6 PM UTC is a reasonable assumption because that's when the job kicks off
      • 2020-10-23 29738, 2020

      • iliekcomputers
        It doesn't dump anything from after when the job kicks off
      • 2020-10-23 29756, 2020

      • iliekcomputers
        The exact timestamp is in the dump name
      • 2020-10-23 29710, 2020

      • pristine___
        ruaok: how often do you listen to massive attack, every week?
      • 2020-10-23 29725, 2020

      • ruaok
        no, not really.
      • 2020-10-23 29748, 2020

      • ruaok
        currently massive attacks is not on rotation for me.
      • 2020-10-23 29729, 2020

      • ruaok
        I've only been listening to it since daily jams feeds it to me.
      • 2020-10-23 29759, 2020

      • ruaok
        https://listenbrainz.org/user/rob/charts?page=1&a… massive attack is in position 24 for me in the past week
      • 2020-10-23 29711, 2020

      • pristine___
        ruaok: is there a way to know when did you last listened to Morcheeba?
      • 2020-10-23 29735, 2020

      • ruaok
        sure. hang on.
      • 2020-10-23 29704, 2020

      • ruaok
      • 2020-10-23 29726, 2020

      • ruaok
        Friday, October 9, 2020 1:00:12 PM
      • 2020-10-23 29750, 2020

      • ruaok
        I did mention that I <3 timescale, yes?
      • 2020-10-23 29755, 2020

      • pristine___
        So you listened to the artist last on 9 October
      • 2020-10-23 29707, 2020

      • ruaok
        yes.
      • 2020-10-23 29705, 2020

      • pristine___
        i will generate the recs again, all the four steps.
      • 2020-10-23 29720, 2020

      • pristine___
        yesterday you only issued request_recommendations
      • 2020-10-23 29726, 2020

      • ruaok
        correct.
      • 2020-10-23 29730, 2020

      • ruaok
        oh, duh. makes sense.
      • 2020-10-23 29734, 2020

      • pristine___
        Yeah
      • 2020-10-23 29740, 2020

      • pristine___
        So the dfs weren't updated
      • 2020-10-23 29755, 2020

      • pristine___
        That's why the recs look constant to me
      • 2020-10-23 29703, 2020

      • pristine___
        I hope this is the only reason
      • 2020-10-23 29712, 2020

      • ruaok
        🤞
      • 2020-10-23 29714, 2020

      • pristine___
        Otherwise there is a big somewhere.
      • 2020-10-23 29718, 2020

      • pristine___
        Bug*
      • 2020-10-23 29723, 2020

      • ruaok
        big bug too
      • 2020-10-23 29705, 2020

      • v6lur has quit
      • 2020-10-23 29732, 2020

      • ruaok
      • 2020-10-23 29715, 2020

      • pristine___
        Commands issued
      • 2020-10-23 29731, 2020

      • pristine___
        I looked at the mails for recs
      • 2020-10-23 29700, 2020

      • pristine___
        They make sense, last generated for history of 8-15 October
      • 2020-10-23 29747, 2020

      • pristine___
        And after that every time we only generated recs (request_recs)
      • 2020-10-23 29753, 2020

      • pristine___
        So yeah
      • 2020-10-23 29753, 2020

      • pristine___
        Yup! Great news :)
      • 2020-10-23 29714, 2020

      • zas
        alastairp: around?
      • 2020-10-23 29713, 2020

      • pristine___
        ruaok: but on a side note, we really should understand the beautiful diff between top/artist and similar artist. Top artist is more of like *giving you a few more tracks of the artists you loved last week*, if a user only listens to 2 artists last week, their playlist will of course be tiny, there is nothing much we can do about it and we should not, because this tiny playlist exactly reflects the user listening