lucifer: damselfish didn't get daily jams today. :(
2022-10-27 30009, 2022
zas
response time seems to degrade, but I'm not sure why yet (it is the response as measured by openresty and stored in logs). We need to investigate this before moving more services to new gateways. For now, I have no explanation.
2022-10-27 30056, 2022
zas
hmmm, something wrong with data representation
2022-10-27 30048, 2022
zas
yup, I know what's happening (more or less), to get correct values (well, almost, since that's a mean of means), be sure to only select rex+rudi in host selector
2022-10-27 30012, 2022
zas
when comparing this way, no significant difference (and that's expected)
(well, around, because DNS TTL is 5 minutes, so that's progressive)
2022-10-27 30022, 2022
atj
OK, so mean request time on kiki was ~40ms and rex/rudi it seems to have settled ~25ms
2022-10-27 30052, 2022
atj
but kiki was handling much more traffic
2022-10-27 30036, 2022
zas
yes
2022-10-27 30027, 2022
zas
traffic is very low on switched websites, we'll have better figures when we'll switch coverartarchive (and mb ofc)
2022-10-27 30052, 2022
atj
will be interesting to see given we have an extra layer on the new gateways
2022-10-27 30008, 2022
zas
according to my measurements, extra layers should have minimal impact: we have the load balancer, then haproxy then openresty, and they use proxy protocol (which means an extra step and lower mtu)
2022-10-27 30026, 2022
zas
but all this is very fast (few ms)
2022-10-27 30045, 2022
atj
and the servers are faster
2022-10-27 30055, 2022
zas
yes, and much more scalable
2022-10-27 30008, 2022
lucifer
mayhem: i see, why that happened. jukevox, alastair etc also didn't get daily jams for same reason 🤦
2022-10-27 30051, 2022
zas
on https traffic (which is cpu intensive) new setup should be much better overall
One thing missing is moving keydb to rex/rudi (they use kiki/herb ones still)
2022-10-27 30021, 2022
atj
can we start putting docker volumes in /srv or something going forward?
2022-10-27 30018, 2022
zas
yes, especially openresty
2022-10-27 30021, 2022
atj
too many containers with volumes in /home/zas ;)
2022-10-27 30033, 2022
zas
yup, I agree
2022-10-27 30036, 2022
Lotheric has quit
2022-10-27 30026, 2022
Lotheric joined the channel
2022-10-27 30000, 2022
zas
I'll not switch more domains until we set up missing parts, the idea here was to get some real traffic in order to detect potential issues
2022-10-27 30040, 2022
atj
it's a good idea to do it this way
2022-10-27 30025, 2022
atj
I'm not sure about minio, stuff like this gives me pause "MinIO strongly recommends production clusters consist of a minimum of 4 minio server nodes in a Server Pool for proper high availability and durability guarantees."
2022-10-27 30055, 2022
atj
but then the alternatives look worse
2022-10-27 30025, 2022
atj
building a proper MinIO cluser is going to cost €€€
2022-10-27 30014, 2022
mayhem
alastairp: monkey : can you please reply to the last.fm rec/UX doodle soon?
2022-10-27 30014, 2022
monkey
Yus
2022-10-27 30020, 2022
zas
atj: yes, I'm not sure either
2022-10-27 30039, 2022
monkey
I tried opening it yesterday but it wasn't workign for me, will try again
2022-10-27 30044, 2022
lucifer
mayhem: i am around now as well to discuss similar recordings.
I was playing with the similarities last night and noticed a few things.
2022-10-27 30037, 2022
mayhem
First off, the 7 day data is pretty bad.
2022-10-27 30054, 2022
lucifer
i see
2022-10-27 30058, 2022
mayhem
which makes little sense, because the first dataset you had a few days ago was even smaller and it was... kinda good.
2022-10-27 30007, 2022
mayhem
so, I can't explain that. might've been a fluke.
2022-10-27 30031, 2022
mayhem
the next realization is that I had an insight, but that insight hasn't been tested yet.
2022-10-27 30049, 2022
mayhem
namely, if we train on too much data, that our data set tends towards noise.
2022-10-27 30058, 2022
mayhem
I dont know if that is actually true.
2022-10-27 30039, 2022
lucifer
we can train on 180-365 days to test that?
2022-10-27 30041, 2022
mayhem
so, I would like to train data sets on: 30d, 90d, 180d, 1year, 3 year, 10 year, all time data sets.
2022-10-27 30049, 2022
lucifer
sounds good
2022-10-27 30006, 2022
lucifer
but we'll have to use appropriate thresholds to keep lookups fast
2022-10-27 30010, 2022
mayhem
and then my goal is to pick a number of tracks as seed tracks and study them.
2022-10-27 30037, 2022
mayhem
yes, that is a good point. can we make the thresholds relative to the amount of data?
2022-10-27 30042, 2022
mayhem
so that it picks itself in a way?
2022-10-27 30007, 2022
lucifer
hmm not sure. at max 100 or 200 similar recordings for each track maybe?
2022-10-27 30000, 2022
mayhem
do you do any pruning of the data while you are generating it or is the pruning one of the last steps?
2022-10-27 30010, 2022
lucifer
last step.
2022-10-27 30053, 2022
mayhem
then yes, lets say we pick max 200 tracks for the next round.
2022-10-27 30003, 2022
mayhem
and then we remove the threshold concept?
2022-10-27 30014, 2022
mayhem
well, swap these concepts?
2022-10-27 30028, 2022
mayhem
I think total count is a better way of doing this.
2022-10-27 30042, 2022
lucifer
i am thinking to use both. 200 max but only if the track meets the threshold, say 5 or 10.
2022-10-27 30057, 2022
mayhem
can we please try without threshold first?
2022-10-27 30012, 2022
mayhem
I would like to see what that is like. it might work better.
2022-10-27 30030, 2022
mayhem
because right now I want to see what the long tail for some of this data is.
2022-10-27 30011, 2022
lucifer
sure. can try.
2022-10-27 30030, 2022
lucifer
so 200 tracks per recording without threshold?
2022-10-27 30053, 2022
mayhem
yes.
2022-10-27 30059, 2022
lucifer
👍
2022-10-27 30003, 2022
mayhem
great.
2022-10-27 30019, 2022
mayhem
another q: are hated tracks filtered post CF data generation?
2022-10-27 30003, 2022
lucifer
nope currently we don't do anything with loved/hated tracks
2022-10-27 30030, 2022
mayhem
but that would be the right place to do this, no?
2022-10-27 30046, 2022
lucifer
yes right.
2022-10-27 30011, 2022
mayhem
let me put that on your todo list as something to do when you have some time.
2022-10-27 30027, 2022
mayhem
at least the hated bit -- that doesn't involve another round of CF tuning.
2022-10-27 30028, 2022
lucifer
where should we do it? spark, LB ingest, LB query time?
2022-10-27 30040, 2022
lucifer
3 degrees of responsiveness
2022-10-27 30008, 2022
mayhem
I was thinking post CF generation. take all the generated tracks from CF and remove all the ones that appear in the user's hated list.
2022-10-27 30014, 2022
mayhem
but I am not married to that approach.
2022-10-27 30040, 2022
mayhem
I think a more comprehensive approach is to tune the inputs to CF so that CF knows all about this.
2022-10-27 30048, 2022
lucifer
yes makes sense, let's start there and see where to go from there.
2022-10-27 30002, 2022
mayhem
but I am hesitant to get into a round of CF filtering right now. it seems to be working and I feel like focusing on other things.
2022-10-27 30006, 2022
mayhem
great!
2022-10-27 30058, 2022
alastairp
mayhem: done. I put a maybe on tues because it's holiday here, not sure what I'll be doing
2022-10-27 30015, 2022
mayhem
seems that we're leaning towards thursday anyway.
2022-10-27 30019, 2022
mayhem
thanks
2022-10-27 30044, 2022
mayhem
yep, thursday it is. will send meeting invite later.
2022-10-27 30048, 2022
alastairp
ok
2022-10-27 30053, 2022
mayhem
alastairp: lucifer : right now the cover art stuff is in a separate repo (initial goals where unclear) and is being served from a different URL.
2022-10-27 30011, 2022
mayhem
these endpoints might get a lot of traffic in the future -- not sure.
2022-10-27 30042, 2022
mayhem
should this code be integrated into the LB codebase and simply be made a new endpoint or two or should we start a new docker container?
2022-10-27 30047, 2022
mayhem
integrate now., move later?
2022-10-27 30007, 2022
mayhem
zas: atj_mb: you two about for this question?
2022-10-27 30010, 2022
alastairp
I think integrate now/move later is better
2022-10-27 30020, 2022
alastairp
is the python code fast?
2022-10-27 30022, 2022
lucifer
sounds good to put in LB for now. when we want to publicize it or show it on lb.org etc, move
2022-10-27 30028, 2022
mayhem
its a question about future growth and I wonder how the new gateways impact this.
2022-10-27 30054, 2022
alastairp
keeping it in the same codebase but starting a new container for it should be straightforward too
2022-10-27 30057, 2022
mayhem
alastairp: there isn't much slow stuff in the code, but it does need to make requests to the DB to resolve IDs.
2022-10-27 30032, 2022
mayhem
and with the new gateways it should be easy to slice one endpoint and have it go to a new server, or?
2022-10-27 30048, 2022
mayhem
I'll proceed with a PR into the codebase for now. loads easier.
2022-10-27 30014, 2022
lucifer
mayhem, i created daily jams now for all users who didn't jams yesterday.
2022-10-27 30022, 2022
mayhem
thank you!
2022-10-27 30045, 2022
atj
o/
2022-10-27 30035, 2022
atj
"cover art stuff" being the code to create the grids?
2022-10-27 30052, 2022
alastairp
yeah, we already send specific URLs/domains to different containers in the gateways (like websockets)
2022-10-27 30003, 2022
Hellow1 joined the channel
2022-10-27 30008, 2022
mayhem
atj: yes.
2022-10-27 30029, 2022
mayhem
atj: this is a general traffic growth question.
2022-10-27 30002, 2022
mayhem
does it ever matter for us to move heavy traffic things to a separate sub-domain or should we always assume that we can do per-URL level routing to back-ends?
2022-10-27 30004, 2022
atj
right, well the new gateways should be able to handle a lot more traffic, however I don't think the gateways are the bottleneck generally
2022-10-27 30007, 2022
mayhem
can we?
2022-10-27 30057, 2022
mayhem
agreed -- this convo is more about the user's perspective and not having to move redirect to new subdomains for when traffic grows.
