Okay. Thanks to both of you for the detailed help :)
2020-05-20 14146, 2020
ruaok
iliekcomputers: on the latest_listen_ts bug, if you can't find a reason where that is used, don't sweat it. the view uses it and it is rather important, but I think the template/react component may not care anymore.
2020-05-20 14106, 2020
ruaok
just stop it.
2020-05-20 14100, 2020
ruaok
most of what we discussed where not mistakes. these are matters of taste, design and consistency.
2020-05-20 14107, 2020
iliekcomputers
ruaok: yeah, the ticket is on my radar, but i agree that it isn't very high priority.
2020-05-20 14134, 2020
ruaok
iliekcomputers: yeah, my point was don't spend too much time on it looking for something that may not be there.
2020-05-20 14142, 2020
ruaok
and don't rush for it.
2020-05-20 14142, 2020
shivam-kapila
okay :). back to work.
2020-05-20 14100, 2020
iliekcomputers
yep, sounds good.
2020-05-20 14129, 2020
ruaok was finally able to reproduce the duplicates on import bug in a small test set
2020-05-20 14158, 2020
ruaok
gah. dyslexia. again. š«
2020-05-20 14147, 2020
iliekcomputers
ishaanshah[m]: hey
2020-05-20 14113, 2020
ishaanshah[m]
Hi!
2020-05-20 14115, 2020
iliekcomputers
How are you?
2020-05-20 14145, 2020
ishaanshah[m]
Done with the PR finally :D
2020-05-20 14153, 2020
iliekcomputers
how are you?
2020-05-20 14104, 2020
iliekcomputers
Oh awesome
2020-05-20 14114, 2020
iliekcomputers
I'll take a look after this.
2020-05-20 14105, 2020
ishaanshah[m]
There are some extra whitespace changes
2020-05-20 14101, 2020
iliekcomputers
Oh, no worries.
2020-05-20 14113, 2020
ishaanshah[m]
Sorry for those, slipped through my sight
2020-05-20 14132, 2020
iliekcomputers
To deploy this, I'll deploy and calculate stats to make sure it works.
2020-05-20 14146, 2020
iliekcomputers
Then truncate the table and calculate again
2020-05-20 14107, 2020
iliekcomputers
That should be enough, unless I'm missing something.
2020-05-20 14136, 2020
ishaanshah[m]
Matrix acting up again
2020-05-20 14144, 2020
ishaanshah[m]
I shoud switch to Irccloud ig
2020-05-20 14146, 2020
ishaanshah[m]
Yeah that sounds reasonable
2020-05-20 14153, 2020
ishaanshah[m]
So we can be sure everything is working
2020-05-20 14114, 2020
iliekcomputers
if you give ruaok your email, he'll add you to the MeB irccloud account. is that okay, ruaok ?
2020-05-20 14121, 2020
ruaok
yep.
2020-05-20 14132, 2020
ruaok
in pm, please.
2020-05-20 14151, 2020
iliekcomputers
ishaanshah[m]: cool, i sifted through the PR once, I don't see anything standing out really. I'll try to merge and deploy today.
2020-05-20 14110, 2020
iliekcomputers
there might be a small thing here or there, but I'll just fix it up myself.
2020-05-20 14129, 2020
iliekcomputers
are you gonna start on the release thing next?
2020-05-20 14138, 2020
iliekcomputers
oh wait, the API endpoint first
2020-05-20 14139, 2020
iliekcomputers
right?
2020-05-20 14104, 2020
ishaanshah[m]
Oh that would be great, I will pm, thanks :D
2020-05-20 14114, 2020
ishaanshah[m]
Cool that sounds good, thanks
2020-05-20 14129, 2020
ishaanshah[m]
I think API should be done tomorrow
2020-05-20 14133, 2020
ishaanshah[m]
Atleast the implementation
2020-05-20 14135, 2020
ishaanshah[m]
Tests maybe day after tomorrow
2020-05-20 14146, 2020
iliekcomputers
ok, awesome.
2020-05-20 14156, 2020
iliekcomputers
i don't really have anything else, do you have something you wanna talk about?
2020-05-20 14102, 2020
ishaanshah[m]
No thats about it
2020-05-20 14122, 2020
iliekcomputers
cool then, i'll get back to testing the PR
2020-05-20 14112, 2020
alastairp
iliekcomputers: how far are you in your investigation of json-schema and other things?
2020-05-20 14122, 2020
ishaanshah[m]
Sure, if there is any major change please let me know
2020-05-20 14122, 2020
iliekcomputers
have not started yet.
2020-05-20 14128, 2020
alastairp
right
2020-05-20 14138, 2020
iliekcomputers
alastairp: i'm gonna use it to formalize the schema for our jsonb columns
2020-05-20 14153, 2020
iliekcomputers
in postgres
2020-05-20 14104, 2020
iliekcomputers
i think
2020-05-20 14105, 2020
alastairp
shivam-kapila: (I'm going to use your code as an example, but don't make any changes based on this discussion until I make a final decision with iliekcomputers)
therefore helping us with typing, field names, and making sure we know what we're dealing with
2020-05-20 14148, 2020
alastairp
I don't want to push too many new crazy things, but it seems like LB is in a great position with lots of movement, a handful of devs, and python 3
2020-05-20 14118, 2020
iliekcomputers
i definitely want typing.
2020-05-20 14133, 2020
alastairp
what I'm thinking: a db module like this defines the get methods, but also some data classes which are used for the outputs of these methods (and maybe the inputs too?)
2020-05-20 14134, 2020
ruaok
+1 to typing, -1 to models.
2020-05-20 14150, 2020
alastairp
use some libraries which help us serialise dicts from postgres into the class
2020-05-20 14113, 2020
alastairp
ruaok: right. for clarification, do you think of "model" as ORM-style stuff, or any class used to store data?
2020-05-20 14140, 2020
ruaok
-100 to ORM, -1 other classes.
2020-05-20 14148, 2020
bitmap
yvanzo: zas: ruaok: I'm going to restart the pg standby on pink with shm-size=2GB and see if that helps with these report queries. shouldn't interupt any services
2020-05-20 14150, 2020
alastairp
right. I'm not recommending ORM stuff ehre.
2020-05-20 14106, 2020
ruaok
bitmap: ok
2020-05-20 14124, 2020
iliekcomputers
I'm +1 to classes, given we do it right, but i don't think we'll be able to do it right now though. i don't have the capacity to do it, and i'd rather keep ishaanshah on the stats stuff for now.
2020-05-20 14142, 2020
alastairp
I'm thinking back to what happened a while back when you were looking at the listen stuff, and we had these to_json/from_json methods that were out of sync, and no one had no idea what fields should be in a dict
2020-05-20 14111, 2020
ruaok
and that happened inside of a class!
2020-05-20 14131, 2020
alastairp
I agree that there's a fine line here
2020-05-20 14111, 2020
alastairp
and I think it's pretty true that many of us "saw the light" when we realised that we could just throw a dict around instead of having to model a class with a whole bunch of fields, and types, and ORM and....
2020-05-20 14118, 2020
iliekcomputers
classes done right can be really amazing for dev productivity.
2020-05-20 14158, 2020
alastairp
but my counterpoint is that every time you throw a dict around, you have to look up 16 different places to make sure that you're reading it in the right way and naming your fields properly
2020-05-20 14114, 2020
alastairp
OK, it sounds like there's general interest in a concept, but perhaps it needs a bit more thought about how to implement it
2020-05-20 14120, 2020
iliekcomputers
i do think this requires more thought though
2020-05-20 14124, 2020
iliekcomputers
++
2020-05-20 14125, 2020
ruaok
+1 to that sentiment about things not being clear
2020-05-20 14139, 2020
alastairp
I'll leave it here, and see if I can come up with some ideas about how to do this with the pybrainzes
2020-05-20 14148, 2020
alastairp
ruaok: do you know about dataclasses? new in recent pythons
alastairp: ah, no. but that does look rather useful and very lightweight. +1 to that.
2020-05-20 14148, 2020
alastairp
right. this is all I'm proposing :)
2020-05-20 14130, 2020
alastairp
cool. I'm not going to burden shivam-kapila with it, but maybe I'll work with iliekcomputers for a while to see how much we can do with this
2020-05-20 14141, 2020
ishaanshah[m]
ruaok: Did you get my PM, not sure if it worked correctly
2020-05-20 14148, 2020
ruaok
no
2020-05-20 14135, 2020
yvanzo
bitmap: maybe we should consider other parameters too
2020-05-20 14136, 2020
ruaok
/msg ruaok hey, dipshit
2020-05-20 14139, 2020
ruaok
should work.
2020-05-20 14105, 2020
ishaanshah[m]
I am on matrix, so not sure if that works
2020-05-20 14110, 2020
bitmap
yvanzo: I'm open to trying if you have suggestions
2020-05-20 14127, 2020
bitmap
2GB does seem to allow the RecordingsWithVaryingTrackLengths query to complete at least
2020-05-20 14139, 2020
ishaanshah[m]
Should I mail you
2020-05-20 14143, 2020
ishaanshah[m]
?
2020-05-20 14145, 2020
ruaok
just send me mail then. rob@metabra... org
2020-05-20 14158, 2020
yvanzo
bitmap: 2GB might just not be enough either, āthe amount of data we want to share between processes there could be quite a bit larger than anything we'd feel comfortable nailing down in the permanent shared memory segmentā https://www.postgresql-archive.org/dynamic-sharedā¦
2020-05-20 14101, 2020
bitmap
of course this with it being the only query running on the server, so not competing with anything else
2020-05-20 14109, 2020
ishaanshah[m]
Cool, thanks :)
2020-05-20 14144, 2020
yvanzo
would probable be nice to restrain parallelism, didn't find how yet
2020-05-20 14124, 2020
bitmap
I mean, there are some configuration options to change how it calculates the parallel cost, but we also have way more than enough memory to let it do its thing
2020-05-20 14150, 2020
ruaok
ishaanshah[m]: done
2020-05-20 14122, 2020
yvanzo
bitmap: maybe go directly for a larger value if we can afford it.
2020-05-20 14110, 2020
ishaanshah[m]
Thanks a lot š
2020-05-20 14113, 2020
bitmap
I wouldn't do less than 8GB to start tbh
2020-05-20 14155, 2020
BrainzGit
[listenbrainz-server] paramsingh merged pull request #856 (masterā¦time_range_spark): LB-575: Add support for more time ranges for "Top Artists" (Spark and DB Insert) https://github.com/metabrainz/listenbrainz-serverā¦
iliekcomputers: Were you able to make it work. If not try removing listenbrainz_postgres volume.
2020-05-20 14108, 2020
iliekcomputers
yeah, i fixed it, no worries.
2020-05-20 14132, 2020
bitmap
outside of docker shmmax is basically unlimited and pg is the only real container we run on floyd
2020-05-20 14150, 2020
bitmap
it's different on pink but still, 8GB isn't much
2020-05-20 14128, 2020
shivam-kapila
> cool. I'm not going to burden shivam-kapila with it, but maybe I'll work with iliekcomputers for a while to see how much we can do with this
2020-05-20 14128, 2020
shivam-kapila
alastairp: I like the concept but I need some opinion on standardizing the class. There are two types of outputs that we can expect from the feedback. Let me paste bin that
2020-05-20 14154, 2020
yvanzo
pg upgrade requiring twice as much memory is quite a big change ^^
2020-05-20 14138, 2020
alastairp
shivam-kapila: yeah, it's definitely the case that in some situations we'll have to take a step back and think about the data that we want, and the situations in which we'll use it
2020-05-20 14115, 2020
bitmap
yvanzo: it doesn't though :P
2020-05-20 14141, 2020
bitmap
I doubt it's even using much more memory overall to solve these queries, it's just hitting an arbitrary limit on shared memory
Here are the cases which I have in mind and the third one is inspired by what you proposed
2020-05-20 14140, 2020
iliekcomputers
ishaanshah[m]: looks like it worked!
2020-05-20 14142, 2020
yvanzo
bitmap: parallel queries do: āparallel queries may consume very substantially more resources than non-parallel queries, because each worker process is a completely separate process which has roughly the same impact on the system as an additional user sessionā https://www.postgresql.org/docs/12/runtime-configā¦
WIP if anyone wants to try the youtube player on LB ^
2020-05-20 14110, 2020
bitmap
yvanzo: I'm not sure that translates to pg requires x times more memory total in practice tho, otherwise our memory usage graphs for floyd would look a lot different
2020-05-20 14145, 2020
alastairp
shivam-kapila: right. it depends on what we want to do with this data. I think this is a good thing, as it makes us think about why we need this data
2020-05-20 14111, 2020
bitmap
of course I don't fully understand how pg uses memory internally, but
2020-05-20 14117, 2020
alastairp
in this case, I think my first suggestion would be to just have two different classes to represent this data. because it's actually different
and it's a good motivation - imagine if we made a mistake and called the wrong method in the website. if we had 2 types of returned data, we'd actually get notifified in the view or other method where we tried to use it
2020-05-20 14118, 2020
bitmap
looks like the amount of active/used mem even decreased slightly since the 18th
