yvanzo The performance of queryResultCache is good for most collections but for few, reading https://solr.apache.org/guide/solr/latest/configu… "Caches are cleared after a commit and usually need to be re-populated before their benefit can be seen again. To counteract this, caches can be "warmed" before a new searcher is considered opened by automatically populating the new cache with values from the old
2025-05-27 14711, 2025
zas[m]
cache."
2025-05-27 14758, 2025
zas[m]
I checked default config for solr 9.7.0 (/opt/solr/releases/solr-9.7.0/server/solr/configsets/_default/conf/solrconfig.xml):... (full message at <https://matrix.chatbrainz.org/_matrix/media/v3/download/chatbrainz.org/zmzIIvUlpemBJspFDIpJqvDL>)
2025-05-27 14714, 2025
zas[m]
So autowarmCount = 0, it means no autowarm, perhaps we should try to enable it for release collections and see if it is better
2025-05-27 14759, 2025
zas[m]
for small collections, it seems cache performance is rather good, even 100% hit ratio (for series collection for example), but for release & release-group we reach max 5%
2025-05-27 14736, 2025
zas[m]
the question is whether we can improve that or not, either by increasing cache size and/or use autowarming
2025-05-27 14742, 2025
zas[m]
I'm not sure how it is configured rigth now, we have per-collection configsets in /etc/solr, but https://solr.apache.org/guide/solr/latest/configu… says solr cloud use zk upconfig for that. Ping me when you are around, I'd like us to test if tweaking autowarm leads to any change
I think the performance is bad for collections that change often (the cache is cleared after each commit with current config), hence the question about autowarm (=keeping old entries after a commit)
2025-05-27 14739, 2025
Sintharu joined the channel
2025-05-27 14741, 2025
Sintharu has quit
2025-05-27 14730, 2025
Maxr1998_ joined the channel
2025-05-27 14738, 2025
Maxr1998 has quit
2025-05-27 14707, 2025
Sintharu joined the channel
2025-05-27 14709, 2025
Sintharu has quit
2025-05-27 14741, 2025
mamanullah7[m] joined the channel
2025-05-27 14741, 2025
mamanullah7[m]
hey lucifer this is db schema which i'll be adding for funkwhale is there any changes required?... (full message at <https://matrix.chatbrainz.org/_matrix/media/v3/download/chatbrainz.org/ZVXLdEcbGKzdOKhtZiHEAVAT>)
2025-05-27 14746, 2025
mamanullah7[m]
* hey lucifer this is db schema which i'll be adding for funkwhale is there any changes required?... (full message at <https://matrix.chatbrainz.org/_matrix/media/v3/download/chatbrainz.org/pxHvHBVaaUnWcQNZNcIcqVfC>)
2025-05-27 14706, 2025
mamanullah7[m]
* hey lucifer this is db schema which i'll be adding for funkwhale is there any changes required?... (full message at <https://matrix.chatbrainz.org/_matrix/media/v3/download/chatbrainz.org/qfnINEqhxgnYyhVvPepxofEt>)
2025-05-27 14735, 2025
lucifer[m]
[@m.amanullah7:matrix.org](https://matrix.to/#/@m.amanullah7:matrix.org) this looks about right. as for the documentation about what part exactly?
2025-05-27 14747, 2025
Sintharu joined the channel
2025-05-27 14731, 2025
suvid[m] joined the channel
2025-05-27 14731, 2025
suvid[m] uploaded an image: (58KiB) < https://matrix.chatbrainz.org/_matrix/media/v3/download/matrix.org/zQFXlHwzbAzWznrBgkwpeXZh/image.png >
2025-05-27 14731, 2025
suvid[m]
Hi lucifer... (full message at <https://matrix.chatbrainz.org/_matrix/media/v3/download/chatbrainz.org/nvlZDHePetvwHFbSWjTGZizt>)
2025-05-27 14731, 2025
suvid[m]
this is the zip file structure for reference
2025-05-27 14733, 2025
suvid[m]
* is the extended streaming history zip file
2025-05-27 14759, 2025
mamanullah7[m]
lucifer: documentation like any standard format to follow like particular table should be added in particular files or in existing once
2025-05-27 14700, 2025
mamanullah7[m]
like I'll add this via migration so should I directly create a new funkwale.py in /db/funckwhale.py
2025-05-27 14736, 2025
suvid[m]
also, where should I make the utility function to upload files?
2025-05-27 14736, 2025
suvid[m]
should i put it in the misc directory?
2025-05-27 14726, 2025
lucifer[m]
m.amanullah7: for migrations, you will have to create a SQL file here. it should contain any new SQL commands that you added to the various SQL files. if you are modifying something existing, there should be appropriate `ALTER` commands for that as well.
2025-05-27 14750, 2025
lucifer[m]
as for the documentation, there is nothing specific as such but I would suggest to look at what the existing services are doing.
2025-05-27 14701, 2025
mamanullah7[m]
Okay sure thanks
2025-05-27 14738, 2025
lucifer[m]
suvid: the code to upload zip files can go in `views` for now i think or in `api_tools`, don't worry too much about it for now. we can discuss the exact location for it later. the ZIP file should be stored by the backend at a certain location and then you should create a task entry in the `background_tasks` table. store the path of this file along with the `user_id` in the metadata of that task. then add a handler in
2025-05-27 14738, 2025
lucifer[m]
`background_tasks` (https://github.com/metabrainz/listenbrainz-server…) to handle the zip file processing. the processing code itself can go in a separate file in that directory. I think you should use a regex to determine the order in which the files should be processed. and process them in earliest to newest order. don't try to combine
<mamanullah7[m]> "hey lucifer this is db schema..." <- hey lucifer while going through existing db implementation i find `error_message` columns to store error msg in existing tables should i also add this for better error handling when imports fail etc.
