alastairp: Re: loosing one's native language: The more languages you add in the mix, the worst it gets.
2021-04-12 10230, 2021
Mr_Monkey
_lucifer: That looks about right at first glance (seems like we do the same thing as with the spotify user), but I haven't looked at the PR yet so I'm not sure.
2021-04-12 10230, 2021
Mr_Monkey
We previously talked about app-wide props that we could use react context for (in another PR) so I assume this could change a bit in the future anyway as we refactor it.
2021-04-12 10214, 2021
_lucifer
Yeah, I want to make it some way which works for both youtube and spotify. currently we have to pass both separetely everywhere.
[listenbrainz-server] amCap1712 opened pull request #1384 (master…youtube-db): Add SQL scripts for connecting new music services and migrating user data https://github.com/metabrainz/listenbrainz-server…
2021-04-12 10258, 2021
_lucifer
alastairp, ruaok, Mr_Monkey: i wanted to deploy the implementation and test on lb test or beta so would be nice to get the db changes in before the rest of the PR.
ruaok: yeah, people were laughing at the custom decode json instruction but it turns out when you can do micro optimisations like that then you get incredible performance/W in return
2021-04-12 10200, 2021
alastairp
I guess hardware video decode has been a thing for yonks now
2021-04-12 10213, 2021
ruaok
not I. our use of ujson makes it clear why this is no laughing matter.
2021-04-12 10234, 2021
alastairp
it'll be interesting to see if this will affect language and data design decisions in ~20 years
2021-04-12 10245, 2021
ruaok
yea and hardware video decode was the first obvious task. speech recognition and ML are next.
2021-04-12 10217, 2021
alastairp
if you have a dedicated low power core to evaluate tensorflow models, what does that make tensorflow?
2021-04-12 10242, 2021
ruaok
a lot more accessible?
2021-04-12 10255, 2021
alastairp
everyone's going to stick with the minimal compatible featureset to be able to continue to take advantage of custom silicon
2021-04-12 10200, 2021
alastairp
even if something better comes along?
2021-04-12 10220, 2021
alastairp
though perhaps that's no different from x86_64 -> x86 -> 80486 -> 8088
2021-04-12 10237, 2021
alastairp
hacks on top of hacks in order to keep supporting the past
2021-04-12 10241, 2021
ruaok
I love RISC V and that people can now dial up custom CPUs and ship them off to TSMC and presto make their own customer systems, often as SoC.
2021-04-12 10245, 2021
ruaok
super exciting.
2021-04-12 10252, 2021
alastairp
if you have a new hypothetical video codec that is half the size of h264, but you need to decode it in software instead of hardware, where's the tradeoff? will we just get complacent with what the CPUs can do and stop advancing?
2021-04-12 10255, 2021
ruaok
*custom
2021-04-12 10227, 2021
ruaok
I doubt it. I see this as an evolution that you cannot ignore.
2021-04-12 10202, 2021
ruaok
classic CPUs -> CPUS + floating point -> Multi core CPU -> Multi function, multi core CPU.
2021-04-12 10217, 2021
alastairp
yep, that's definitely true too
2021-04-12 10228, 2021
alastairp
got sse4? use it, otherwise emulate down to sse2
2021-04-12 10242, 2021
ruaok
its the 2nd evolutionary step past moore
2021-04-12 10236, 2021
ruaok
and given that more cores are not necessarily going to give increased performance because of software bottlenecks, custom CPUs have new and exciting possibilities for performance improvement.
2021-04-12 10248, 2021
ruaok
ignoring this trend would be fatal.
2021-04-12 10214, 2021
ruaok
heck, even the RPi now has 4 cores. my watch has 2. RPi pico has two for crying out loud!
Jeff Geerling's pi hacks have been super cool. what you can do with such a small soc is neat
2021-04-12 10242, 2021
alastairp
ruaok: yeah, I did see it when it came up
2021-04-12 10217, 2021
ruaok
its something we need to look at before too long. I believe our performance for our python sites is quite shit too.
2021-04-12 10222, 2021
alastairp
though from what I can tell it seems to be a "Hey, we have a cool async web framework that is _soooo_ much faster than flask"
2021-04-12 10252, 2021
alastairp
I mean, this isn't making the python code itself any faster. and this is why we spin up 300 workers on uwsgi anyway
2021-04-12 10232, 2021
ruaok
I'm thining longer term with more scalability. I think a low hanging fruit is to try pypy
2021-04-12 10233, 2021
alastairp
my gut feel is it's one of those "this is a problem if you want everything to be perfect, but if 'meh, let's throw more uwsgis at the problem' is good enough for you then it's not really a practical issue"
2021-04-12 10244, 2021
alastairp
but I've not considered it in too much detail
2021-04-12 10211, 2021
ruaok
before too long I want to try running our web containers in pypy and if it is possible and what kind of change we can see.
2021-04-12 10223, 2021
alastairp
it's possible that if we get to the point where we need to set up a second web server, we could instead look at more detailed optimisations and see if that gets us some breathing room and how much
2021-04-12 10236, 2021
ruaok
yes, agreed.
2021-04-12 10214, 2021
ruaok
though, I think one usage pattern that is going to be different from last.fm vs LB is that many listens will come from services, rather than time critical WS requests.
2021-04-12 10235, 2021
alastairp
yeah, for sure
2021-04-12 10251, 2021
ruaok
in essence that makes the optimization of the spotify reader, which does not use a web framework key for us.
2021-04-12 10207, 2021
ruaok
and of course the ingestion pipeline which we know needs optimizing.
2021-04-12 10236, 2021
alastairp
oh, that reminds me
2021-04-12 10252, 2021
alastairp
I saw that _lucifer opened a ticket about the ujson upgrade
I think I found out why we got so much confusion with "are we using timestamps or dates"
2021-04-12 10246, 2021
alastairp
because the current ujson that we have transparently converts python datetime objects to timestamps
2021-04-12 10230, 2021
ruaok
where do we use custom serialization?
2021-04-12 10232, 2021
alastairp
and I suspect at some time we had a specific class -> json deserialiser that took care of this, but at one point in time we forgot to use it and ujson magically converted it to timestamp
2021-04-12 10246, 2021
ruaok
for PG maybe?
2021-04-12 10221, 2021
ruaok
my gripes with timestamps in LB is that we have both datetime and epoch timestamps.
