[@mayhem:chatbrainz.org](https://matrix.to/#/@mayhem:chatbrainz.org) lb and meb redis are separate already. not sure what caused it. were meb or lb under elevated traffic when this happened?
I would check myself but I am unsure when this started.
FWIW, I did find some queries to donor api just before the redis mess started
mayhem[m]
no elevated traffic; nothing I could see, except from redis connection messages.
lucifer[m]
So I have implemented a cache for those queries
Testing it in beta at the moment.
mayhem[m]
and redis flushing its data to disk with heavy IO once a minute. that may not seem like a good idea...
lucifer[m]
Those queries could create a lot of db connections to meb db bringing down meb too
mayhem[m]
ah, interesting.
lucifer[m]
As for redis flushing, the bulk of the data in redis would be metadata caches.
Wild guess would be metadata caches overwhelmed redis somehow and that caused issues in lb which caused the meb dono queries to lag and brought down meb too
But I will have to dig deeper
To be sure
zas[m]
mayhem: How metabrainz.org linked to kiss? Kiss exhibits high disk write activity every 4 minutes for 3 minutes, leading to overall high disk I/O, load is rather low, cpu usage is low, memory usage looks normal
mayhem[m]
That might be redis flushing to disk, which seems excessive
<zas[m]> "But metabrainz.org use this..." <- no, but it looks like lucifer may have put a lot of load on the DB testing a pesky query, which may have caused this. that query is now being cached to limit the impact.
mayhem, zas: the default redis config that we are using for LB saves the existing RDB to memory on exit and loads it back in memory on restart. it also apparently keeps track of the max memory usage that was reached in the last lifetime and then allocates the max needed memory again. we had only 900M of data in redis but it occupied 42 G because at some point in the last year lb-redis on kiss used 42G. large amount of memory also
means more CPU consumption. i manually got rid of the existing dumps and then stopped the redis server with nosave to prevent it from writing the dump to disk. the cpu usage has now dropped to almost 1% and the memory usage is of course a few MBs at the moment, i expect that to grow to 1G or something in a few hours as normal usage.
* last year,, * so lb-redis on, * used 42G ever since even after restarts. large
mayhem[m]
does that mean we had a lot of data that never expired?
lucifer[m]
i don't think this is related to yesterday's issue but still should speed up our redis use in LB.
no the data expired but redis doesn't return the memory to the OS>
i would expect it to know that 40/42G are unallocated and don't need to be seached and hence not affect CPU usage but don't know enough about redis internals to tell.
mayhem[m]
thats less than ideal.
lucifer[m]
that's how most modern allocators work but we can modify the config to add a max-memory limit on how much it can allocate.
we can also look into redis-compatible alternatives for caching. i guess that was the plan anyway ever since they changed their licensing.
i don't think we will face this issue again anytime soon though, the 42G of memory usage was most likely from the redis-streams test to replace rabbitmq for spark last year.
mayhem[m]
great. thanks for digging into this!
monkey[m]
<_BrainzGit> "[listenbrainz-server] amCap1712..." <- I suppose we could cache this query result for more than 10 minutes (maybe 30m?), but not sure how much impact that would have.
Jade[m]
<lucifer[m]> "we can also look into redis-..." <- Iirc they changed it back recently
However the forks are still going, so there's a bit of an ecosystem split
<lucifer[m]> "mayhem: redis logs are unavailab..." <- I briefly checked the container last night and have some truncated logs from my terminal that I can send (including `MONITOR` output)
lucifer[m]
bitmap: thanks, tool a look at it but didn't find anything abnormal.
bitmap[m]
yeah, I didn't either but thanks for taking a look!
lucifer[m]
<monkey[m]> "I suppose we could cache this..." <- if we invalidated the cache when a new donation is received we could cache it longer. but anyway for now the impact is how long LB takes to detect a new donation and then enable flair.
monkey[m]
lucifer[m]: Yeah, I made the same calculation, but figured we get on average one or two donations from LB users a day, so though maybe we could be more aggressive with the caching if needed.
lucifer[m]
yup makes sense
monkey[m]
Well, that's going to be an interesting merge experience...
monkey[m] uploaded an image: (8KiB) < https://matrix.chatbrainz.org/_matrix/media/v3/download/chatbrainz.org/VycMfUDszNayMaFRnAPoCiiK/image.png >
lucifer[m]
oof
monkey[m]
13 conflicted files. Not too bad actually
lucifer[m]
<Jade[m]> "Iirc they changed it back..." <- oh i see, i wasn't aware of that. nice
