bitmap: remind me, how hard would it be to make the change to the work types and whatnot in the pots so that they have the type name as the ID?
2020-10-23 29741, 2020
reosarevok
(I still have a ton of tabs opened to change type names but should probably wait until that is done)
2020-10-23 29713, 2020
sumedh has quit
2020-10-23 29736, 2020
sumedh joined the channel
2020-10-23 29748, 2020
v6lur joined the channel
2020-10-23 29728, 2020
adhawkins
Morning all. Just thought I'd report back about my dockerised MBserver setup. I was getting regular OOM errors from it. Since increasing the RAM to 6 GB nearly 16 days ago I haven't had a single one.
2020-10-23 29721, 2020
adhawkins
Seems a lot, considering the previous VM that ran it worked fine with just 4 Gig. I guess if the docker-compose file can be arranged so as to optionally not start the search indexing stuff that might help?
2020-10-23 29702, 2020
Gazooo79494 has quit
2020-10-23 29741, 2020
Gazooo79494 joined the channel
2020-10-23 29706, 2020
ruaok
mooooin!
2020-10-23 29726, 2020
ruaok
pristine___: on #1153, that is mostly moving code around, not so much making functional changes, right?
2020-10-23 29735, 2020
ruaok wonders how closely he should ready this PR
2020-10-23 29752, 2020
ruaok
but I am glad that you made a separate PR for the refactoring work. đź‘Ť
2020-10-23 29738, 2020
pristine___
ruaok: yeah, have just dumped dataframe function in a utils so that create_df_recording and create_df_artist can use it from there
2020-10-23 29757, 2020
ruaok
ok, then it should be fine. let me have one more look.
I think there be a few more (1 or 2) refactor PRs before I start with create_df_artist.
2020-10-23 29721, 2020
pristine___
Then we are good to go!
2020-10-23 29754, 2020
ruaok
seems about right. I'm excited for the refactorings because that will make more models possible.
2020-10-23 29701, 2020
ruaok
user-user would be pretty sexy.
2020-10-23 29709, 2020
ruaok
and I'm dreaming up all sorts of things that could be good models too. I'm also stoked that creatings recs is now fast. ⛄
2020-10-23 29711, 2020
ruaok
alastairp: do you know how to discover and run unit tests from inside a python script?
2020-10-23 29728, 2020
alastairp
hi
2020-10-23 29736, 2020
alastairp
what's your final goal?
2020-10-23 29702, 2020
ruaok
test.sh has `py.test`in it, but given how we're setting up running troi via model invocation, I'd like to add that as one of the top level commands in cli.py
promising. I was trying it via unittest, but I now see that that was the wrong level.
2020-10-23 29757, 2020
alastairp
right, unittest has test discovery stuff that you can do, but I think that keeping it at pytest is good
2020-10-23 29709, 2020
ruaok
pytest.main(["-x", "."])
2020-10-23 29712, 2020
ruaok
does the trick
2020-10-23 29719, 2020
alastairp
there are some interesting ways of writing tests in pytest, which I like
2020-10-23 29742, 2020
alastairp
keep in mind that it'd be good to be able to pass args to pytest - specifically I use -v often, and also passing a single test to it
2020-10-23 29759, 2020
ruaok
I was expecting that. :) on it.
2020-10-23 29755, 2020
alastairp
just related to your PR, do you prefer travis, or you just did it because it was easy? Want me to set it up on jenkins, or...?
2020-10-23 29722, 2020
alastairp
it seems like almost all of our checks are on jenkins atm
2020-10-23 29728, 2020
ruaok
honestly, I am confused by our setup. why do we have two? is one preferred over the other?
2020-10-23 29735, 2020
_lucifer
do docker's new policies affect meb?
2020-10-23 29742, 2020
alastairp
_lucifer: yes
2020-10-23 29747, 2020
ruaok
_lucifer: what are they?
2020-10-23 29749, 2020
alastairp
I have similar questions
2020-10-23 29704, 2020
alastairp
I guess we've had jenkins for a long time,
2020-10-23 29705, 2020
_lucifer
they are limiting number of pulls and image retention period for free accounts
2020-10-23 29706, 2020
ruaok
similar questions about our test setup or docker?
2020-10-23 29712, 2020
alastairp
about test setup
2020-10-23 29702, 2020
alastairp
I don't specifically know why some tests on some projects use travis. I believe that jenkins existed at the time that they were added, but I can't be sure
2020-10-23 29703, 2020
ruaok
zas: ^^ see comment about docker changes. can you please examine if this will impact us and what we might be able to do to mitigate it?
I believe bitmap moved some MB tests from travis to jenkins recently
2020-10-23 29733, 2020
ruaok
alastairp: well, if you feel like setting another CI, I don't much care. I'll get the test running reliably and well from the command line in the scope of this PR
2020-10-23 29718, 2020
alastairp
I don't care strongly either, so I guess we're at an impasse :)
2020-10-23 29722, 2020
alastairp
we can leave it, that's fine
2020-10-23 29733, 2020
ruaok
"Free plan – anonymous users: 100 pulls per 6 hours "
2020-10-23 29746, 2020
ruaok
lets leave it until we have a clear indication as to why/what we should do.
2020-10-23 29712, 2020
ruaok
_lucifer: given that we user our own images internally and that others dont pull them, this may not affect us.
2020-10-23 29727, 2020
kepstin has quit
2020-10-23 29727, 2020
tripathyprateek[ has quit
2020-10-23 29728, 2020
goldenshimmer has quit
2020-10-23 29729, 2020
MagnusSvensson[m has quit
2020-10-23 29734, 2020
dotmavriq[m] has quit
2020-10-23 29739, 2020
maxlath[m] has quit
2020-10-23 29746, 2020
reg[m] has quit
2020-10-23 29747, 2020
joshuaboniface has quit
2020-10-23 29736, 2020
_lucifer
đź‘Ť
2020-10-23 29745, 2020
ruaok
alastairp: I'll just keep pushing new tests to the fix-basic-tests branch. tell me when you're close to reviewing it so I don't push more stuff in your way.
2020-10-23 29739, 2020
alastairp
ok. I'm not sure how much time I have today (on another project today), but if you think it's small I can probably have a quick look later after lunch
ruaok: you too artist recs this week look similar to last week and a week before. Is that?
2020-10-23 29728, 2020
pristine___
Top*
2020-10-23 29748, 2020
ruaok
I havent been paying attention to top artists as much -- similar gives better results for daily jams, so not sure.
2020-10-23 29702, 2020
ruaok
similar artists are vastly more diverse this week than last.
2020-10-23 29726, 2020
ruaok
I sort of suspect that was related to ishaanshah loading more listens in a giant batch. that probably threw things off.
2020-10-23 29713, 2020
pristine___
Yeah, top artist are not diverse because let's say a user listens to 10 artist last week, so tracks of only 10 artists in recs, but tracks of 10 * 10 artists in similar tracks
2020-10-23 29730, 2020
pristine___
ruaok: are dumps back on track?
2020-10-23 29756, 2020
ruaok
the PR to fix them was just merged, so we should see regular imports very soon.
2020-10-23 29717, 2020
ruaok
hmm. daily-jams for top artists only gives 5 tracks.
2020-10-23 29718, 2020
ishaanshah
pristine___: yes they are
2020-10-23 29710, 2020
ruaok
yeah, my top artists is so massive attack heavy, that the artist limiter tosses out most of the tracks.
2020-10-23 29750, 2020
ruaok
perhaps you are right and need to include some artist limiting at the CF level.
2020-10-23 29701, 2020
pristine___
ishaanshah: can you tell me what is the latest ts of listens in spark cluster? I mean is there a to know that?
2020-10-23 29712, 2020
reg[m] joined the channel
2020-10-23 29712, 2020
joshuaboniface joined the channel
2020-10-23 29712, 2020
maxlath[m] joined the channel
2020-10-23 29712, 2020
kepstin joined the channel
2020-10-23 29713, 2020
goldenshimmer joined the channel
2020-10-23 29713, 2020
tripathyprateek[ joined the channel
2020-10-23 29713, 2020
MagnusSvensson[m joined the channel
2020-10-23 29743, 2020
pristine___
ruaok: I would really like to improve top/similar on spark side once I am done with feedback stuff
2020-10-23 29745, 2020
ishaanshah
pristine___: you will have to download latest dump manually and check
2020-10-23 29704, 2020
pristine___
Phew, rough idea?
2020-10-23 29706, 2020
ishaanshah
Incremental dump
2020-10-23 29738, 2020
ishaanshah
Most prolly yesterday 6-8pm UTC
2020-10-23 29749, 2020
pristine___
thanks!
2020-10-23 29726, 2020
iliekcomputers
6 PM UTC is a reasonable assumption because that's when the job kicks off
2020-10-23 29738, 2020
iliekcomputers
It doesn't dump anything from after when the job kicks off
2020-10-23 29756, 2020
iliekcomputers
The exact timestamp is in the dump name
2020-10-23 29710, 2020
pristine___
ruaok: how often do you listen to massive attack, every week?
2020-10-23 29725, 2020
ruaok
no, not really.
2020-10-23 29748, 2020
ruaok
currently massive attacks is not on rotation for me.
2020-10-23 29729, 2020
ruaok
I've only been listening to it since daily jams feeds it to me.
They make sense, last generated for history of 8-15 October
2020-10-23 29747, 2020
pristine___
And after that every time we only generated recs (request_recs)
2020-10-23 29753, 2020
pristine___
So yeah
2020-10-23 29753, 2020
pristine___
Yup! Great news :)
2020-10-23 29714, 2020
zas
alastairp: around?
2020-10-23 29713, 2020
pristine___
ruaok: but on a side note, we really should understand the beautiful diff between top/artist and similar artist. Top artist is more of like *giving you a few more tracks of the artists you loved last week*, if a user only listens to 2 artists last week, their playlist will of course be tiny, there is nothing much we can do about it and we should not, because this tiny playlist exactly reflects the user listening
