#metabrainz

/

      • nelgin has quit
      • sumedh joined the channel
      • reosarevok
        bitmap: remind me, how hard would it be to make the change to the work types and whatnot in the pots so that they have the type name as the ID?
      • (I still have a ton of tabs opened to change type names but should probably wait until that is done)
      • sumedh has quit
      • sumedh joined the channel
      • v6lur joined the channel
      • adhawkins
        Morning all. Just thought I'd report back about my dockerised MBserver setup. I was getting regular OOM errors from it. Since increasing the RAM to 6 GB nearly 16 days ago I haven't had a single one.
      • Seems a lot, considering the previous VM that ran it worked fine with just 4 Gig. I guess if the docker-compose file can be arranged so as to optionally not start the search indexing stuff that might help?
      • Gazooo79494 has quit
      • Gazooo79494 joined the channel
      • ruaok
        mooooin!
      • pristine___: on #1153, that is mostly moving code around, not so much making functional changes, right?
      • ruaok wonders how closely he should ready this PR
      • but I am glad that you made a separate PR for the refactoring work. 👍
      • pristine___
        ruaok: yeah, have just dumped dataframe function in a utils so that create_df_recording and create_df_artist can use it from there
      • ruaok
        ok, then it should be fine. let me have one more look.
      • great first step!
      • should I merge it? (it looks good to me)
      • pristine___
        Yeah, sure
      • Thanks!
      • BrainzGit
        [listenbrainz-server] mayhem merged pull request #1153 (master…dataframe-utils): Utils for shareable dataframe functions https://github.com/metabrainz/listenbrainz-serv...
      • pristine___
        I think there be a few more (1 or 2) refactor PRs before I start with create_df_artist.
      • Then we are good to go!
      • ruaok
        seems about right. I'm excited for the refactorings because that will make more models possible.
      • user-user would be pretty sexy.
      • and I'm dreaming up all sorts of things that could be good models too. I'm also stoked that creatings recs is now fast. ⛄
      • alastairp: do you know how to discover and run unit tests from inside a python script?
      • alastairp
        hi
      • what's your final goal?
      • ruaok
        test.sh has `py.test`in it, but given how we're setting up running troi via model invocation, I'd like to add that as one of the top level commands in cli.py
      • python -m troi.cli test
      • to run tests and get rid of test.sh
      • alastairp
      • ruaok tries it
      • ruaok
        promising. I was trying it via unittest, but I now see that that was the wrong level.
      • alastairp
        right, unittest has test discovery stuff that you can do, but I think that keeping it at pytest is good
      • ruaok
        pytest.main(["-x", "."])
      • does the trick
      • alastairp
        there are some interesting ways of writing tests in pytest, which I like
      • keep in mind that it'd be good to be able to pass args to pytest - specifically I use -v often, and also passing a single test to it
      • ruaok
        I was expecting that. :) on it.
      • alastairp
        just related to your PR, do you prefer travis, or you just did it because it was easy? Want me to set it up on jenkins, or...?
      • it seems like almost all of our checks are on jenkins atm
      • ruaok
        honestly, I am confused by our setup. why do we have two? is one preferred over the other?
      • _lucifer
        do docker's new policies affect meb?
      • alastairp
        _lucifer: yes
      • ruaok
        _lucifer: what are they?
      • alastairp
        I have similar questions
      • I guess we've had jenkins for a long time,
      • _lucifer
        they are limiting number of pulls and image retention period for free accounts
      • ruaok
        similar questions about our test setup or docker?
      • alastairp
        about test setup
      • I don't specifically know why some tests on some projects use travis. I believe that jenkins existed at the time that they were added, but I can't be sure
      • ruaok
        zas: ^^ see comment about docker changes. can you please examine if this will impact us and what we might be able to do to mitigate it?
      • _lucifer
      • alastairp
        I believe bitmap moved some MB tests from travis to jenkins recently
      • ruaok
        alastairp: well, if you feel like setting another CI, I don't much care. I'll get the test running reliably and well from the command line in the scope of this PR
      • alastairp
        I don't care strongly either, so I guess we're at an impasse :)
      • we can leave it, that's fine
      • ruaok
        "Free plan – anonymous users: 100 pulls per 6 hours "
      • lets leave it until we have a clear indication as to why/what we should do.
      • _lucifer: given that we user our own images internally and that others dont pull them, this may not affect us.
      • kepstin has quit
      • tripathyprateek[ has quit
      • goldenshimmer has quit
      • MagnusSvensson[m has quit
      • dotmavriq[m] has quit
      • maxlath[m] has quit
      • reg[m] has quit
      • joshuaboniface has quit
      • _lucifer
        👍
      • ruaok
        alastairp: I'll just keep pushing new tests to the fix-basic-tests branch. tell me when you're close to reviewing it so I don't push more stuff in your way.
      • alastairp
        ok. I'm not sure how much time I have today (on another project today), but if you think it's small I can probably have a quick look later after lunch
      • ruaok
      • like that, right?
      • alastairp: the PR is still small. maybe I should just stop adding to it in hopes that you can look after lunch.
      • I'll do that.
      • dotmavriq[m] joined the channel
      • alastairp
        perfect to both those points, thaks
      • actually, I lied. I've got some time booked with zas at 2 to do some granfa stuff, so I do have time to do some reviews :)
      • ruaok: fyi, will deploy a branch to LB beta at 2-ish for an hour or so for testing
      • ruaok
        I'll have the branch ready for review before 2 then.
      • k
      • d4rkie joined the channel
      • D4RK-PH0ENiX has quit
      • alastairp: https://github.com/metabrainz/troi-recommendati... is ready. plenty long as it is.
      • pristine___
        ruaok: you too artist recs this week look similar to last week and a week before. Is that?
      • Top*
      • ruaok
        I havent been paying attention to top artists as much -- similar gives better results for daily jams, so not sure.
      • similar artists are vastly more diverse this week than last.
      • I sort of suspect that was related to ishaanshah loading more listens in a giant batch. that probably threw things off.
      • pristine___
        Yeah, top artist are not diverse because let's say a user listens to 10 artist last week, so tracks of only 10 artists in recs, but tracks of 10 * 10 artists in similar tracks
      • ruaok: are dumps back on track?
      • ruaok
        the PR to fix them was just merged, so we should see regular imports very soon.
      • hmm. daily-jams for top artists only gives 5 tracks.
      • ishaanshah
        pristine___: yes they are
      • ruaok
        yeah, my top artists is so massive attack heavy, that the artist limiter tosses out most of the tracks.
      • perhaps you are right and need to include some artist limiting at the CF level.
      • pristine___
        ishaanshah: can you tell me what is the latest ts of listens in spark cluster? I mean is there a to know that?
      • reg[m] joined the channel
      • joshuaboniface joined the channel
      • maxlath[m] joined the channel
      • kepstin joined the channel
      • goldenshimmer joined the channel
      • tripathyprateek[ joined the channel
      • MagnusSvensson[m joined the channel
      • ruaok: I would really like to improve top/similar on spark side once I am done with feedback stuff
      • ishaanshah
        pristine___: you will have to download latest dump manually and check
      • pristine___
        Phew, rough idea?
      • ishaanshah
        Incremental dump
      • Most prolly yesterday 6-8pm UTC
      • pristine___
        thanks!
      • iliekcomputers
        6 PM UTC is a reasonable assumption because that's when the job kicks off
      • It doesn't dump anything from after when the job kicks off
      • The exact timestamp is in the dump name
      • pristine___
        ruaok: how often do you listen to massive attack, every week?
      • ruaok
        no, not really.
      • currently massive attacks is not on rotation for me.
      • I've only been listening to it since daily jams feeds it to me.
      • https://listenbrainz.org/user/rob/charts?page=1... massive attack is in position 24 for me in the past week
      • pristine___
        ruaok: is there a way to know when did you last listened to Morcheeba?
      • ruaok
        sure. hang on.
      • Friday, October 9, 2020 1:00:12 PM
      • I did mention that I <3 timescale, yes?
      • pristine___
        So you listened to the artist last on 9 October
      • ruaok
        yes.
      • pristine___
        i will generate the recs again, all the four steps.
      • yesterday you only issued request_recommendations
      • ruaok
        correct.
      • oh, duh. makes sense.
      • pristine___
        Yeah
      • So the dfs weren't updated
      • That's why the recs look constant to me
      • I hope this is the only reason
      • ruaok
        🤞
      • pristine___
        Otherwise there is a big somewhere.
      • Bug*
      • ruaok
        big bug too
      • v6lur has quit
      • pristine___
        Commands issued
      • I looked at the mails for recs
      • They make sense, last generated for history of 8-15 October
      • And after that every time we only generated recs (request_recs)
      • So yeah
      • Yup! Great news :)
      • zas
        alastairp: around?
      • pristine___
        ruaok: but on a side note, we really should understand the beautiful diff between top/artist and similar artist. Top artist is more of like *giving you a few more tracks of the artists you loved last week*, if a user only listens to 2 artists last week, their playlist will of course be tiny, there is nothing much we can do about it and we should not, because this tiny playlist exactly reflects the user listening