oh there's also the option to compress the cache with zstd instead of gzip.
2022-10-19 29253, 2022
lucifer
in your experience with MLHD stuff, is zstd faster than gzip?
2022-10-19 29228, 2022
alastairp
yes, do it
2022-10-19 29244, 2022
lucifer
compression level?
2022-10-19 29203, 2022
alastairp
~7 is a pretty good compromise
2022-10-19 29208, 2022
alastairp
are we looking for speed or size?
2022-10-19 29217, 2022
lucifer
speed
2022-10-19 29226, 2022
alastairp
I guess that's the tradeoff - speed means that building is faster but pushing the cache is slower
2022-10-19 29238, 2022
lucifer
note that the action will have to download/upload the cache as well
2022-10-19 29241, 2022
lucifer
right
2022-10-19 29227, 2022
alastairp
zstd -3 is ~the same compression level as gzip but a million times faster
2022-10-19 29259, 2022
lucifer
i see.
2022-10-19 29213, 2022
alastairp
once you go up to ~ -9 you probably need threads to keep up with the speed. I don't know how many cores are in the workers, or if you can set that parameter
previously we used `user.id`, I think it makes sense here that `user.musicbrainz_username` should be public too, along with user_ref. email + license_choice if confidental
2022-10-19 29201, 2022
lucifer
alastairp: ah i see. yes makes sense.
2022-10-19 29226, 2022
lucifer
musicbrainz_username is public everywhere so makes sense to be public here as well.
2022-10-19 29216, 2022
alastairp
yeah, exactly. it's been private for at least 6 years, but not sure why that decision was initially made.
alastairp: wow! do you have an idea of what is being stored in kafka? I thought postgres was the primary data store and these were "unprocessed" events
2022-10-19 29215, 2022
alastairp
nope, no idea what's in there. I agree, I think that pg should be the primary store too
2022-10-19 29246, 2022
lucifer
what's the retention policy of the kafka instance?
2022-10-19 29203, 2022
alastairp
I'm looking through the logs and can't find any clear indication about what happened. So I'm not sure if it's the config items we added, or the cleanup that you did, or it goes and re-packs db files in the background?
2022-10-19 29214, 2022
alastairp
lucifer: theoretically it was always configured to 24h
2022-10-19 29215, 2022
lucifer
events aren't deleted asap after processing but only when the retention policy is met afaik.
2022-10-19 29232, 2022
alastairp
but yesterday we tried to add a "no more than 50gb" limit too (I later lowered it down to 40gb)
2022-10-19 29211, 2022
alastairp
but I'm not completely sure how these limits work when both of them are in place, or exactly how kafka splits the db up for doing cleanups
2022-10-19 29248, 2022
alastairp
also unsure about when it decides to delete something in respect to when it starts up. the forum posts that bitmap and I found seemed to indicate that it'd start pruning "immediately", but it took about 4h from when I restarted it yesterday
2022-10-19 29202, 2022
alastairp
and no log informations about which policy triggered this operation in the end (size or time)
2022-10-19 29239, 2022
Hellow1 has quit
2022-10-19 29256, 2022
lucifer
alastairp: i see. iiuc, that limit is still per topic. i am interpreting the output of kafka-configs in container correctly.
thanks for the mountains of code needed to get this feature done. Even Kirsten (user Damselfish) thanks you -- she is quite excited to get this feature going.
clearing the setting makes sense to me. what other options do you see?
2022-10-19 29241, 2022
lucifer
we could do nothing, the access token is removed on disconnecting so it would become null in the query. the benefit of not doing anything is that if the user reconnects then they won't have to enable this again.
2022-10-19 29212, 2022
lucifer
and changing permissions from say record only listens to play only or both. or any other combination involves a disconnect.
2022-10-19 29256, 2022
mayhem
Not doing anything, so that the user can disconnect, then reconnect and not lose out sounds best.
2022-10-19 29209, 2022
Hellow2 joined the channel
2022-10-19 29211, 2022
lucifer
👍
2022-10-19 29233, 2022
mayhem
can we check the missing access token before doing the troi work to generate a playlist?
2022-10-19 29255, 2022
mayhem
carrying out the wolrk and then not being able to deliver it would be wasteful.
2022-10-19 29222, 2022
lucifer
yes can do that. but we still want to generate a playlist in any case, no? just that for null token don't attempt to upload
2022-10-19 29248, 2022
lucifer
*don't attempt to upload to spotify but still upload to LB.
2022-10-19 29251, 2022
mayhem
yes, good point. thats all good then.
2022-10-19 29219, 2022
Hellow2
Hi :) Thank's to all that answered my questions. I have been able to completly implement the tagger today.. So thanks <33
2022-10-19 29238, 2022
mayhem
great to hear that!
2022-10-19 29250, 2022
mayhem didn't know the world needed another tagger in 2022.
2022-10-19 29210, 2022
Hellow2
yea actually not, but it is a complete music cli downloader cuz I am tired of stuff that I listen to not being on spotify xD The tagger is just a part of it
2022-10-19 29236, 2022
mayhem
ah, I see. good to hear.
2022-10-19 29243, 2022
Hellow2
yep :D
2022-10-19 29206, 2022
mayhem
I've been talking about the growing backlash against spotify and how people are embracing their music collections again. a very good development, if you ask me.
2022-10-19 29239, 2022
mayhem
we're clearly keen to get people to be more engaged with music and if spotify is making people numb to music, then by all means, lets change and improve!
2022-10-19 29205, 2022
Hellow2
I've really agree. Though I don't see many people that don't use spotify. To listen to much music you need lots of storage, you need to sync you're library between phone and laptop. The syncing may be easy with scp and simmilar but the storage can be a huge problem. Also the affort of building a good enough library are probaply far to high for the masses to ditch spotify.
Hi lucifer, riksucks, Pratha-Fish , yellowhatpro , ansh , Shubh, I’ll be kn Delhi starting tomorrow for the next few weeks. I have your MetaBrainz summit t-shirts and lots of metabrainz stickers for you guys! Please connect with me on how you would like to collect them. I would prefer we all meet at one place on a day, chat around about MetaBrainz, have fun and use this as an excuse for a get together!
