#metabrainz

bitmap: remind me, how hard would it be to make the change to the work types and whatnot in the pots so that they have the type name as the ID?

(I still have a ton of tabs opened to change type names but should probably wait until that is done)

Morning all. Just thought I'd report back about my dockerised MBserver setup. I was getting regular OOM errors from it. Since increasing the RAM to 6 GB nearly 16 days ago I haven't had a single one.

Seems a lot, considering the previous VM that ran it worked fine with just 4 Gig. I guess if the docker-compose file can be arranged so as to optionally not start the search indexing stuff that might help?

mooooin!

pristine___: on #1153, that is mostly moving code around, not so much making functional changes, right?

but I am glad that you made a separate PR for the refactoring work. 👍

ruaok: yeah, have just dumped dataframe function in a utils so that create_df_recording and create_df_artist can use it from there

ok, then it should be fine. let me have one more look.

great first step!

should I merge it? (it looks good to me)

Yeah, sure

Thanks!

[listenbrainz-server] mayhem merged pull request #1153 (master…dataframe-utils): Utils for shareable dataframe functions https://github.com/metabrainz/listenbrainz-serv...

I think there be a few more (1 or 2) refactor PRs before I start with create_df_artist.

Then we are good to go!

seems about right. I'm excited for the refactorings because that will make more models possible.

user-user would be pretty sexy.

and I'm dreaming up all sorts of things that could be good models too. I'm also stoked that creatings recs is now fast. ⛄

alastairp: do you know how to discover and run unit tests from inside a python script?

hi

what's your final goal?

test.sh has `py.test`in it, but given how we're setting up running troi via model invocation, I'd like to add that as one of the top level commands in cli.py

python -m troi.cli test

to run tests and get rid of test.sh

maybe https://docs.pytest.org/en/stable/usage.html#ca... ?

promising. I was trying it via unittest, but I now see that that was the wrong level.

right, unittest has test discovery stuff that you can do, but I think that keeping it at pytest is good

pytest.main(["-x", "."])

does the trick

there are some interesting ways of writing tests in pytest, which I like

keep in mind that it'd be good to be able to pass args to pytest - specifically I use -v often, and also passing a single test to it

I was expecting that. :) on it.

just related to your PR, do you prefer travis, or you just did it because it was easy? Want me to set it up on jenkins, or...?

it seems like almost all of our checks are on jenkins atm

honestly, I am confused by our setup. why do we have two? is one preferred over the other?

do docker's new policies affect meb?

_lucifer: yes

_lucifer: what are they?

I have similar questions

I guess we've had jenkins for a long time,

they are limiting number of pulls and image retention period for free accounts

similar questions about our test setup or docker?

about test setup

I don't specifically know why some tests on some projects use travis. I believe that jenkins existed at the time that they were added, but I can't be sure

zas: ^^ see comment about docker changes. can you please examine if this will impact us and what we might be able to do to mitigate it?

https://www.docker.com/blog/scaling-dockers-bus... and https://www.docker.com/blog/docker-hub-image-re...

I believe bitmap moved some MB tests from travis to jenkins recently

alastairp: well, if you feel like setting another CI, I don't much care. I'll get the test running reliably and well from the command line in the scope of this PR

I don't care strongly either, so I guess we're at an impasse :)

we can leave it, that's fine

"Free plan – anonymous users: 100 pulls per 6 hours "

_lucifer: given that we user our own images internally and that others dont pull them, this may not affect us.

👍

alastairp: I'll just keep pushing new tests to the fix-basic-tests branch. tell me when you're close to reviewing it so I don't push more stuff in your way.

ok. I'm not sure how much time I have today (on another project today), but if you think it's small I can probably have a quick look later after lunch

like that, right?

alastairp: the PR is still small. maybe I should just stop adding to it in hopes that you can look after lunch.

I'll do that.

perfect to both those points, thaks

actually, I lied. I've got some time booked with zas at 2 to do some granfa stuff, so I do have time to do some reviews :)

ruaok: fyi, will deploy a branch to LB beta at 2-ish for an hour or so for testing

I'll have the branch ready for review before 2 then.

k

alastairp: https://github.com/metabrainz/troi-recommendati... is ready. plenty long as it is.

ruaok: you too artist recs this week look similar to last week and a week before. Is that?

Top*

I havent been paying attention to top artists as much -- similar gives better results for daily jams, so not sure.

similar artists are vastly more diverse this week than last.

I sort of suspect that was related to ishaanshah loading more listens in a giant batch. that probably threw things off.

Yeah, top artist are not diverse because let's say a user listens to 10 artist last week, so tracks of only 10 artists in recs, but tracks of 10 * 10 artists in similar tracks

ruaok: are dumps back on track?

the PR to fix them was just merged, so we should see regular imports very soon.

hmm. daily-jams for top artists only gives 5 tracks.

pristine___: yes they are

yeah, my top artists is so massive attack heavy, that the artist limiter tosses out most of the tracks.

perhaps you are right and need to include some artist limiting at the CF level.

ishaanshah: can you tell me what is the latest ts of listens in spark cluster? I mean is there a to know that?

ruaok: I would really like to improve top/similar on spark side once I am done with feedback stuff

pristine___: you will have to download latest dump manually and check

Phew, rough idea?

Incremental dump

Most prolly yesterday 6-8pm UTC

thanks!

6 PM UTC is a reasonable assumption because that's when the job kicks off

It doesn't dump anything from after when the job kicks off

The exact timestamp is in the dump name

ruaok: how often do you listen to massive attack, every week?

no, not really.

currently massive attacks is not on rotation for me.

I've only been listening to it since daily jams feeds it to me.

https://listenbrainz.org/user/rob/charts?page=1... massive attack is in position 24 for me in the past week

ruaok: is there a way to know when did you last listened to Morcheeba?

sure. hang on.

Friday, October 9, 2020 1:00:12 PM

I did mention that I <3 timescale, yes?

So you listened to the artist last on 9 October

yes.

i will generate the recs again, all the four steps.

yesterday you only issued request_recommendations

correct.

oh, duh. makes sense.

Yeah

So the dfs weren't updated

That's why the recs look constant to me

I hope this is the only reason

🤞

Otherwise there is a big somewhere.

Bug*

big bug too

wooo, this is great news for us: https://twitter.com/songshift/status/1318916343...

Commands issued

I looked at the mails for recs

They make sense, last generated for history of 8-15 October

And after that every time we only generated recs (request_recs)

So yeah

Yup! Great news :)

alastairp: around?

ruaok: but on a side note, we really should understand the beautiful diff between top/artist and similar artist. Top artist is more of like *giving you a few more tracks of the artists you loved last week*, if a user only listens to 2 artists last week, their playlist will of course be tiny, there is nothing much we can do about it and we should not, because this tiny playlist exactly reflects the user listening