"Silicon Valley types want whatever's next because there might be money in it, but also they are fundamentally not very interested in inhabiting or maintaining the new realities they shape; it's too much like work. Maintaining things is hard, and requires much more care than making things does."
2023-05-18 13819, 2023
aerozol
Monkey: btw I haven't forgotten the things I have to do, I've just had this other deadline 😬
atj: zas: I think we need to think about scalable fast disk solutions for LB. disk space will be a continual problem going forward.
2023-05-18 13855, 2023
atj
Can you clarify what you mean by scalable?
2023-05-18 13838, 2023
mayhem
that when we need more disk, we don't have to go through these amazing machinations of downtime, install disks, migrate, more downtime to remove old disks. that dance is very tiring.
2023-05-18 13813, 2023
mayhem
I suppose baby steps are to have data and OS volumes.
2023-05-18 13841, 2023
mayhem
ideally so that we have 6 drive bays: 2 OS, 2 current data volume, 2 future data volume.
2023-05-18 13854, 2023
mayhem
but even that is a hassle for everytime disks fill up.
2023-05-18 13816, 2023
mayhem
but where else do you run a postgres instance, but your SSD?
2023-05-18 13819, 2023
mayhem
tricky.
2023-05-18 13824, 2023
mayhem
aerozol: yes, I've always felt that friction of "you don't care about the future" anytime I dealt with VC backed companies or with academics.
2023-05-18 13830, 2023
atj
zas will know more about what is feasible from a hardware perspective. I don't know how much flexibility we get with Hetzner.
2023-05-18 13847, 2023
mayhem
they can't see beyond the current quarter or the current funding cycle.
2023-05-18 13817, 2023
atj
NVMEs aren't hot swappable
2023-05-18 13826, 2023
mayhem
one option to explore are these scalable attached storage boxes that you can get for cloud instances. but that doesn't help our bare metal servers.
2023-05-18 13839, 2023
mayhem
exactly. none of these solutions are great.
2023-05-18 13843, 2023
atj
zas will probably not be keen (which is fair enough) but I'd like to utilise ZFS for local storage in certain scenarios. I suspect most of our data is pretty compressible, so transparent compression could drop usage quite significantly.
2023-05-18 13835, 2023
atj
There are also various other benefits which I won't go into.
2023-05-18 13800, 2023
mayhem
on disk compression is not something you want to have live under postgres.
2023-05-18 13820, 2023
mayhem
postgres wants to be as close to bare metal as possible.
2023-05-18 13802, 2023
atj
Postgres is widely used on ZFS and IME you often hit compression ratios of 3:1+ on table data.
2023-05-18 13848, 2023
atj
This actually speeds things up because modern CPUs can compress/ decompress faster than the storage transfer rate
2023-05-18 13859, 2023
atj
At the end of the day storage is pretty cheap, so ideally we should try to work out some 2-3 year growth predictions and then size accordingly
2023-05-18 13829, 2023
mayhem
really? huh.
2023-05-18 13838, 2023
mayhem
however, we're often CPU constrained on our servers, since hetzner never does multi-proc setups.
2023-05-18 13813, 2023
mayhem
but if you say that ZFS can do this, then we should load the LB data on ZFS and give it a shot.
2023-05-18 13829, 2023
atj
We can discuss with zas tomorrow, I'd like to give it a try at least.
2023-05-18 13807, 2023
mayhem
agreed.
2023-05-18 13825, 2023
mayhem
this buys time, but should we hit a sudden growth spurt with LB, we will have disk issues.
2023-05-18 13808, 2023
atj
Even if we have multiple scenarios, it would be a great to try and make some rough 2-3 year growth estimates
I could see three growth scenarios: 1. Continue straight line growth from where we have been (read: slow) 2. accelerated growth where growth rate is 2-10 times current growth. (medium growth) 3. Hockey stick (read: nightmare)
2023-05-18 13805, 2023
mayhem
1 is cake. ZFS might a good fit.
2023-05-18 13823, 2023
mayhem
2. needs a more comprehensive plan, but not thing extravagant.
2023-05-18 13837, 2023
mayhem
3. here be dragons.
2023-05-18 13808, 2023
atj
3 is probably not worth considering too much at this stage, but 2-10x current growth will give a good range of values
2023-05-18 13855, 2023
mayhem
we need to have an answer for 3.
2023-05-18 13825, 2023
mayhem
I mean, LB will never be the next last.fm. last.fm was a runaway success because it was free music streaming for the masses in a world that didn't have streaming.
2023-05-18 13802, 2023
mayhem
now we're drowning in streaming, so we're only ever going to be a smaller player. but that is fine.
2023-05-18 13834, 2023
mayhem
still, a growth rate that lies between 2 and 3 is still troublesome and that is a lot more likely.
2023-05-18 13857, 2023
mayhem
but 2-10x growth is a good goal for the next year, I would say.
