oh there's also the option to compress the cache with zstd instead of gzip.
in your experience with MLHD stuff, is zstd faster than gzip?
alastairp
yes, do it
lucifer
compression level?
alastairp
~7 is a pretty good compromise
are we looking for speed or size?
lucifer
speed
alastairp
I guess that's the tradeoff - speed means that building is faster but pushing the cache is slower
lucifer
note that the action will have to download/upload the cache as well
right
alastairp
zstd -3 is ~the same compression level as gzip but a million times faster
lucifer
i see.
alastairp
once you go up to ~ -9 you probably need threads to keep up with the speed. I don't know how many cores are in the workers, or if you can set that parameter
previously we used `user.id`, I think it makes sense here that `user.musicbrainz_username` should be public too, along with user_ref. email + license_choice if confidental
lucifer
alastairp: ah i see. yes makes sense.
musicbrainz_username is public everywhere so makes sense to be public here as well.
alastairp
yeah, exactly. it's been private for at least 6 years, but not sure why that decision was initially made.
alastairp: wow! do you have an idea of what is being stored in kafka? I thought postgres was the primary data store and these were "unprocessed" events
alastairp
nope, no idea what's in there. I agree, I think that pg should be the primary store too
lucifer
what's the retention policy of the kafka instance?
alastairp
I'm looking through the logs and can't find any clear indication about what happened. So I'm not sure if it's the config items we added, or the cleanup that you did, or it goes and re-packs db files in the background?
lucifer: theoretically it was always configured to 24h
lucifer
events aren't deleted asap after processing but only when the retention policy is met afaik.
alastairp
but yesterday we tried to add a "no more than 50gb" limit too (I later lowered it down to 40gb)
but I'm not completely sure how these limits work when both of them are in place, or exactly how kafka splits the db up for doing cleanups
also unsure about when it decides to delete something in respect to when it starts up. the forum posts that bitmap and I found seemed to indicate that it'd start pruning "immediately", but it took about 4h from when I restarted it yesterday
and no log informations about which policy triggered this operation in the end (size or time)
Hellow1 has quit
lucifer
alastairp: i see. iiuc, that limit is still per topic. i am interpreting the output of kafka-configs in container correctly.
thanks for the mountains of code needed to get this feature done. Even Kirsten (user Damselfish) thanks you -- she is quite excited to get this feature going.
clearing the setting makes sense to me. what other options do you see?
lucifer
we could do nothing, the access token is removed on disconnecting so it would become null in the query. the benefit of not doing anything is that if the user reconnects then they won't have to enable this again.
and changing permissions from say record only listens to play only or both. or any other combination involves a disconnect.
mayhem
Not doing anything, so that the user can disconnect, then reconnect and not lose out sounds best.
Hellow2 joined the channel
lucifer
👍
mayhem
can we check the missing access token before doing the troi work to generate a playlist?
carrying out the wolrk and then not being able to deliver it would be wasteful.
lucifer
yes can do that. but we still want to generate a playlist in any case, no? just that for null token don't attempt to upload
*don't attempt to upload to spotify but still upload to LB.
mayhem
yes, good point. thats all good then.
Hellow2
Hi :) Thank's to all that answered my questions. I have been able to completly implement the tagger today.. So thanks <33
mayhem
great to hear that!
mayhem didn't know the world needed another tagger in 2022.
Hellow2
yea actually not, but it is a complete music cli downloader cuz I am tired of stuff that I listen to not being on spotify xD The tagger is just a part of it
mayhem
ah, I see. good to hear.
Hellow2
yep :D
mayhem
I've been talking about the growing backlash against spotify and how people are embracing their music collections again. a very good development, if you ask me.
we're clearly keen to get people to be more engaged with music and if spotify is making people numb to music, then by all means, lets change and improve!
Hellow2
I've really agree. Though I don't see many people that don't use spotify. To listen to much music you need lots of storage, you need to sync you're library between phone and laptop. The syncing may be easy with scp and simmilar but the storage can be a huge problem. Also the affort of building a good enough library are probaply far to high for the masses to ditch spotify.
Hi lucifer, riksucks, Pratha-Fish , yellowhatpro , ansh , Shubh, I’ll be kn Delhi starting tomorrow for the next few weeks. I have your MetaBrainz summit t-shirts and lots of metabrainz stickers for you guys! Please connect with me on how you would like to collect them. I would prefer we all meet at one place on a day, chat around about MetaBrainz, have fun and use this as an excuse for a get together!
