yvanzo The performance of queryResultCache is good for most collections but for few, reading https://solr.apache.org/guide/solr/latest/confi... "Caches are cleared after a commit and usually need to be re-populated before their benefit can be seen again. To counteract this, caches can be "warmed" before a new searcher is considered opened by automatically populating the new cache with values from the old
So autowarmCount = 0, it means no autowarm, perhaps we should try to enable it for release collections and see if it is better
for small collections, it seems cache performance is rather good, even 100% hit ratio (for series collection for example), but for release & release-group we reach max 5%
the question is whether we can improve that or not, either by increasing cache size and/or use autowarming
I'm not sure how it is configured rigth now, we have per-collection configsets in /etc/solr, but https://solr.apache.org/guide/solr/latest/confi... says solr cloud use zk upconfig for that. Ping me when you are around, I'd like us to test if tweaking autowarm leads to any change
I think the performance is bad for collections that change often (the cache is cleared after each commit with current config), hence the question about autowarm (=keeping old entries after a commit)
lucifer: documentation like any standard format to follow like particular table should be added in particular files or in existing once
like I'll add this via migration so should I directly create a new funkwale.py in /db/funckwhale.py
suvid[m]
also, where should I make the utility function to upload files?
should i put it in the misc directory?
lucifer[m]
m.amanullah7: for migrations, you will have to create a SQL file here. it should contain any new SQL commands that you added to the various SQL files. if you are modifying something existing, there should be appropriate `ALTER` commands for that as well.
as for the documentation, there is nothing specific as such but I would suggest to look at what the existing services are doing.
mamanullah7[m]
Okay sure thanks
lucifer[m]
suvid: the code to upload zip files can go in `views` for now i think or in `api_tools`, don't worry too much about it for now. we can discuss the exact location for it later. the ZIP file should be stored by the backend at a certain location and then you should create a task entry in the `background_tasks` table. store the path of this file along with the `user_id` in the metadata of that task. then add a handler in
`background_tasks` (https://github.com/metabrainz/listenbrainz-serv...) to handle the zip file processing. the processing code itself can go in a separate file in that directory. I think you should use a regex to determine the order in which the files should be processed. and process them in earliest to newest order. don't try to combine
yvanzo: hi, the [missing search indexes dumps](https://community.metabrainz.org/t/20250524-data-dump-for-search-indexes-missing/762296) should be fixed by PR 3550 ^, I'm working on bundling and rsyncing the failed dumps by hand
mamanullah7[m]
<mamanullah7[m]> "hey lucifer this is db schema..." <- hey lucifer while going through existing db implementation i find `error_message` columns to store error msg in existing tables should i also add this for better error handling when imports fail etc.
