iliekcomputers how fast did the stats calculation takes place
2020-05-20 14132, 2020
iliekcomputers
request triggered at 06:28, all messages sent back at 06:36
2020-05-20 14144, 2020
iliekcomputers
the query took 390s.
2020-05-20 14100, 2020
ishaanshah[m]
Nice, thats really fast
2020-05-20 14106, 2020
ishaanshah[m]
Without any optimisation
2020-05-20 14118, 2020
iliekcomputers
i was talking to a friend who's been working with spark.
2020-05-20 14101, 2020
iliekcomputers
he said that a major source for optimization we might wanna look at is how we partition the dataframes between workers when they are loaded into memory
2020-05-20 14129, 2020
iliekcomputers
right now, it is possible that we have a lot of executors sitting idle because the data isn't partitioned minutely enough.
2020-05-20 14102, 2020
alastairp
ruaok: perfect, that' looks great
2020-05-20 14108, 2020
iliekcomputers
anyways, i think we'll cross that bridge when we get to it, but something to investigate.
2020-05-20 14142, 2020
ishaanshah[m]
Hmm, but right now I think the bottleneck is time taken for full export
2020-05-20 14152, 2020
iliekcomputers
yeah.
2020-05-20 14110, 2020
iliekcomputers
that's a pita.
2020-05-20 14127, 2020
iliekcomputers
there's some real low hanging fruit there
2020-05-20 14132, 2020
ishaanshah[m]
We should optimize that or look into streaming listens as they come
2020-05-20 14141, 2020
ishaanshah[m]
But thats after gsoc ig😅
2020-05-20 14109, 2020
iliekcomputers
yeah. i would say let's build on the infrastructure we have right now, and then see what to do.
2020-05-20 14124, 2020
iliekcomputers
i've changed the cron jobs so that a new dump gets imported every week
2020-05-20 14135, 2020
iliekcomputers
so the weekly report thing should still work.
2020-05-20 14143, 2020
ishaanshah[m]
Yeah
2020-05-20 14120, 2020
iliekcomputers
did you see my comment about the API breaking if we change the code that gets the count before the new stats get written into the db?
2020-05-20 14132, 2020
ishaanshah[m]
Yeah I did
2020-05-20 14153, 2020
ishaanshah[m]
If it takes only six minutes to gen stats again
2020-05-20 14113, 2020
ishaanshah[m]
I think it would be ok if API is broken for six mins?
2020-05-20 14129, 2020
iliekcomputers
i guess, it's a question of tradeoff.
2020-05-20 14144, 2020
iliekcomputers
we've said that the endpoint is in beta, so it probably will be ok.
2020-05-20 14103, 2020
iliekcomputers
however, the profile page breaking for 6 min, i wouldn't want to take that downtime if we can avoid it
2020-05-20 14108, 2020
ishaanshah[m]
Or maybe I will just do try/except
2020-05-20 14117, 2020
ishaanshah[m]
And then do a further PR to remove it agin
2020-05-20 14119, 2020
iliekcomputers
so you'd have to write the try-except or if-else anyways.
2020-05-20 14142, 2020
iliekcomputers
i'd just put it in there for both of them and remove it later
2020-05-20 14144, 2020
iliekcomputers
yeah.
2020-05-20 14154, 2020
ishaanshah[m]
Yep, I will do that
2020-05-20 14112, 2020
iliekcomputers
also, i was thinking about something else
2020-05-20 14134, 2020
iliekcomputers
we've made schema changes to the json in the artist column. will the old data get overwritten when new data comes in? It won't linger around because the new data gets written to all_time while the old data was written to something_else.all_time?
2020-05-20 14148, 2020
ishaanshah[m]
I think it should get overwritten
2020-05-20 14112, 2020
ishaanshah[m]
But it would be better to drop the table and then recalculate
2020-05-20 14121, 2020
ishaanshah[m]
Once the new schema is in
2020-05-20 14135, 2020
ishaanshah[m]
It will definitely get overwritten
2020-05-20 14137, 2020
iliekcomputers
hmm, maybe truncating the table is a good idea.
2020-05-20 14149, 2020
iliekcomputers
but if it gets overwritten, that's cool
2020-05-20 14117, 2020
ishaanshah[m]
I believe the count was outside all_time before
2020-05-20 14126, 2020
ishaanshah[m]
So that will remain lingering
2020-05-20 14141, 2020
ishaanshah[m]
But other than that it should be fine
2020-05-20 14143, 2020
iliekcomputers
:/
2020-05-20 14152, 2020
iliekcomputers
yea, we probably should truncate it then
2020-05-20 14131, 2020
ishaanshah[m]
Can we truncate before deployment
2020-05-20 14103, 2020
iliekcomputers
why?
2020-05-20 14116, 2020
ishaanshah[m]
If we are gonna truncate anyways
2020-05-20 14126, 2020
ishaanshah[m]
It doesn't make sense to have try/except
2020-05-20 14103, 2020
iliekcomputers
true.
2020-05-20 14133, 2020
iliekcomputers
if the table doesn't have data, it'll give us a 204.
2020-05-20 14139, 2020
iliekcomputers
yeah, that sounds reasonable.
2020-05-20 14147, 2020
ishaanshah[m]
Yeah
2020-05-20 14153, 2020
iliekcomputers
and the profile page won't break.
2020-05-20 14100, 2020
ishaanshah[m]
I will have a look at the profile page to make sure it doesn't break
shivam-kapila: cool, thanks. I normally drink plain black tea with only water, but I will try an all-milk chai masala today
2020-05-20 14145, 2020
alastairp
iliekcomputers: one thing that we're missing in AB/LB is a local tool for checking flake8 errors (and an easy way to run it)
2020-05-20 14119, 2020
alastairp
I'm not sure I want to go as far as a pre-commit hook, but we should have something as easy as tests for people to get feedback (maybe we even run it as part of tests?)
2020-05-20 14133, 2020
alastairp
so that we can minimise these "fix pep8 errors" commits ;)
2020-05-20 14107, 2020
iliekcomputers
We have a lint.sh that runs prettier fixes. Something similar for Python would be nice, I agree
one thing I don't like about pep8speaks is that it reads config from its own file :( would be nice to not have to duplicate this
2020-05-20 14129, 2020
alastairp
otherwise, I was thinking of doing a single PR in AB soon which just cleans up all issues
2020-05-20 14101, 2020
alastairp
then it'll be guaranteed that any message in a PR will be because of whoever wrote that PR
2020-05-20 14101, 2020
ZaphodBeeblebrox
[20:42] <alastairp> I have a masala chai
2020-05-20 14101, 2020
ZaphodBeeblebrox
[20:41] <alastairp> I need help from someone who drinks chai
2020-05-20 14101, 2020
ZaphodBeeblebrox
alastairp: what I do is I boil water and milk and use about half'n'half
2020-05-20 14101, 2020
ZaphodBeeblebrox
(granted I've also made it with assam-tea and all the spices from scratch :D)
2020-05-20 14131, 2020
alastairp
thanks ZaphodBeeblebrox! It looks like it's "permitted" to try a bunch of variations and see what I like, then
2020-05-20 14153, 2020
shivam-kapila
The milk-water combo is all your prefs
2020-05-20 14105, 2020
iliekcomputers
I'm not really trying to enforce pep8 across the repository. A lot of the whitespace guidelines, for example, are nice to haves, but optional imo. a big PR doesn't really make sense in that context. The benefit of pep8 speaks for us is that it catches bugs sometimes too.
2020-05-20 14126, 2020
alastairp
if that's the case, it might be nice to see if we can disable those warnings, then? I dunno... it seems like we should either go all-in, or get rid of what's not useful for us
2020-05-20 14133, 2020
alastairp
otherwise we grow complacent
2020-05-20 14140, 2020
alastairp
"oh, that's just one of the warnings that we're allowed to ignore"
2020-05-20 14146, 2020
ZaphodBeeblebrox
ruaok> most played artists on listenbrainz and their most played tracks:
2020-05-20 14147, 2020
ZaphodBeeblebrox
IRON MAIDEN! \m/ run to the hills \m/
2020-05-20 14150, 2020
iliekcomputers
Yeah, true.
2020-05-20 14151, 2020
alastairp
I've not used autopep8 before, I'll look into it. I use pycharm to format all of my stuff anyway, so _most_ of the time I don't run into problems
2020-05-20 14109, 2020
iliekcomputers
But making a disable list is work :D
2020-05-20 14128, 2020
alastairp
well, pep8speaks gives a list of codes
2020-05-20 14140, 2020
alastairp
choose the ones you don't want, and add it...
2020-05-20 14144, 2020
ZaphodBeeblebrox
alastairp: well I'm some norwegian cat doing it - granted I read on wikipedia about how it's "properly" done, but still :D
2020-05-20 14116, 2020
ZaphodBeeblebrox
maybe I should try doing it all milk next time I make one 🤔
2020-05-20 14157, 2020
iliekcomputers
alastairp: I was joking, I'll start adding rules we ignore to our config next PR onwards.
2020-05-20 14114, 2020
alastairp
ZaphodBeeblebrox: right. When I was with friends in India it was all milk, so I guess that's what they did in that specific area
2020-05-20 14138, 2020
alastairp
iliekcomputers: great. do you mind coordinating about it? just to make sure we're consistent, and agree on what to ignore?
2020-05-20 14157, 2020
ZaphodBeeblebrox
when I did the boil/soak assam-with-spices method I used only milk i think
2020-05-20 14157, 2020
alastairp
which one do you not like? Indent continuation?
2020-05-20 14128, 2020
ZaphodBeeblebrox
bwahahaha 289181 Johann Sebastian Bach
2020-05-20 14138, 2020
iliekcomputers
The 2 lines after functions, but 1 line after methods
2020-05-20 14146, 2020
ZaphodBeeblebrox
I think ours is the only time any classical - music things have been on such a list
2020-05-20 14149, 2020
iliekcomputers
I don't have any others rn
2020-05-20 14150, 2020
alastairp
oh, I love that one
2020-05-20 14154, 2020
alastairp
:(
2020-05-20 14100, 2020
ZaphodBeeblebrox
<3 us
2020-05-20 14111, 2020
alastairp
I mean, I guess I just got used to it after pycharm complained so much
