bitmap: remind me, how hard would it be to make the change to the work types and whatnot in the pots so that they have the type name as the ID?
(I still have a ton of tabs opened to change type names but should probably wait until that is done)
sumedh has quit
sumedh joined the channel
v6lur joined the channel
adhawkins
Morning all. Just thought I'd report back about my dockerised MBserver setup. I was getting regular OOM errors from it. Since increasing the RAM to 6 GB nearly 16 days ago I haven't had a single one.
Seems a lot, considering the previous VM that ran it worked fine with just 4 Gig. I guess if the docker-compose file can be arranged so as to optionally not start the search indexing stuff that might help?
Gazooo79494 has quit
Gazooo79494 joined the channel
ruaok
mooooin!
pristine___: on #1153, that is mostly moving code around, not so much making functional changes, right?
ruaok wonders how closely he should ready this PR
but I am glad that you made a separate PR for the refactoring work. 👍
pristine___
ruaok: yeah, have just dumped dataframe function in a utils so that create_df_recording and create_df_artist can use it from there
ruaok
ok, then it should be fine. let me have one more look.
I think there be a few more (1 or 2) refactor PRs before I start with create_df_artist.
Then we are good to go!
ruaok
seems about right. I'm excited for the refactorings because that will make more models possible.
user-user would be pretty sexy.
and I'm dreaming up all sorts of things that could be good models too. I'm also stoked that creatings recs is now fast. ⛄
alastairp: do you know how to discover and run unit tests from inside a python script?
alastairp
hi
what's your final goal?
ruaok
test.sh has `py.test`in it, but given how we're setting up running troi via model invocation, I'd like to add that as one of the top level commands in cli.py
promising. I was trying it via unittest, but I now see that that was the wrong level.
alastairp
right, unittest has test discovery stuff that you can do, but I think that keeping it at pytest is good
ruaok
pytest.main(["-x", "."])
does the trick
alastairp
there are some interesting ways of writing tests in pytest, which I like
keep in mind that it'd be good to be able to pass args to pytest - specifically I use -v often, and also passing a single test to it
ruaok
I was expecting that. :) on it.
alastairp
just related to your PR, do you prefer travis, or you just did it because it was easy? Want me to set it up on jenkins, or...?
it seems like almost all of our checks are on jenkins atm
ruaok
honestly, I am confused by our setup. why do we have two? is one preferred over the other?
_lucifer
do docker's new policies affect meb?
alastairp
_lucifer: yes
ruaok
_lucifer: what are they?
alastairp
I have similar questions
I guess we've had jenkins for a long time,
_lucifer
they are limiting number of pulls and image retention period for free accounts
ruaok
similar questions about our test setup or docker?
alastairp
about test setup
I don't specifically know why some tests on some projects use travis. I believe that jenkins existed at the time that they were added, but I can't be sure
ruaok
zas: ^^ see comment about docker changes. can you please examine if this will impact us and what we might be able to do to mitigate it?
I believe bitmap moved some MB tests from travis to jenkins recently
ruaok
alastairp: well, if you feel like setting another CI, I don't much care. I'll get the test running reliably and well from the command line in the scope of this PR
alastairp
I don't care strongly either, so I guess we're at an impasse :)
we can leave it, that's fine
ruaok
"Free plan – anonymous users: 100 pulls per 6 hours "
lets leave it until we have a clear indication as to why/what we should do.
_lucifer: given that we user our own images internally and that others dont pull them, this may not affect us.
kepstin has quit
tripathyprateek[ has quit
goldenshimmer has quit
MagnusSvensson[m has quit
dotmavriq[m] has quit
maxlath[m] has quit
reg[m] has quit
joshuaboniface has quit
_lucifer
👍
ruaok
alastairp: I'll just keep pushing new tests to the fix-basic-tests branch. tell me when you're close to reviewing it so I don't push more stuff in your way.
alastairp
ok. I'm not sure how much time I have today (on another project today), but if you think it's small I can probably have a quick look later after lunch
ruaok: you too artist recs this week look similar to last week and a week before. Is that?
Top*
ruaok
I havent been paying attention to top artists as much -- similar gives better results for daily jams, so not sure.
similar artists are vastly more diverse this week than last.
I sort of suspect that was related to ishaanshah loading more listens in a giant batch. that probably threw things off.
pristine___
Yeah, top artist are not diverse because let's say a user listens to 10 artist last week, so tracks of only 10 artists in recs, but tracks of 10 * 10 artists in similar tracks
ruaok: are dumps back on track?
ruaok
the PR to fix them was just merged, so we should see regular imports very soon.
hmm. daily-jams for top artists only gives 5 tracks.
ishaanshah
pristine___: yes they are
ruaok
yeah, my top artists is so massive attack heavy, that the artist limiter tosses out most of the tracks.
perhaps you are right and need to include some artist limiting at the CF level.
pristine___
ishaanshah: can you tell me what is the latest ts of listens in spark cluster? I mean is there a to know that?
reg[m] joined the channel
joshuaboniface joined the channel
maxlath[m] joined the channel
kepstin joined the channel
goldenshimmer joined the channel
tripathyprateek[ joined the channel
MagnusSvensson[m joined the channel
ruaok: I would really like to improve top/similar on spark side once I am done with feedback stuff
ishaanshah
pristine___: you will have to download latest dump manually and check
pristine___
Phew, rough idea?
ishaanshah
Incremental dump
Most prolly yesterday 6-8pm UTC
pristine___
thanks!
iliekcomputers
6 PM UTC is a reasonable assumption because that's when the job kicks off
It doesn't dump anything from after when the job kicks off
The exact timestamp is in the dump name
pristine___
ruaok: how often do you listen to massive attack, every week?
ruaok
no, not really.
currently massive attacks is not on rotation for me.
I've only been listening to it since daily jams feeds it to me.
They make sense, last generated for history of 8-15 October
And after that every time we only generated recs (request_recs)
So yeah
Yup! Great news :)
zas
alastairp: around?
pristine___
ruaok: but on a side note, we really should understand the beautiful diff between top/artist and similar artist. Top artist is more of like *giving you a few more tracks of the artists you loved last week*, if a user only listens to 2 artists last week, their playlist will of course be tiny, there is nothing much we can do about it and we should not, because this tiny playlist exactly reflects the user listening