mamanullah7[m] uploaded an image: (213KiB) < https://matrix.chatbrainz.org/_matrix/media/v3/download/matrix.org/kCyWxFvwopbiQsVXoYOhUczO/Screenshot%202025-05-22%20at%201.22.57%E2%80%AFPM.png >
2025-05-22 14253, 2025
mamanullah7[m]
lucifer: i did somehow 😒 but even after generating fake data also i'm getting `An unexpected error occurred.`... (full message at <https://matrix.chatbrainz.org/_matrix/media/v3/download/chatbrainz.org/FIdZGnDtNhfQfcxaZpHIRsaC>)
2025-05-22 14212, 2025
mamanullah7[m]
* lucifer: i did somehow 😒 but even after generating fake data also i'm getting `An unexpected error occurred.`... (full message at <https://matrix.chatbrainz.org/_matrix/media/v3/download/chatbrainz.org/IKyqdJTeiaNaoAGXwkiHBRoC>)
2025-05-22 14234, 2025
nobiz joined the channel
2025-05-22 14252, 2025
adhawkins[m]
Trying it now. I do get the same collation messages while the reindex is going on. Will monitor the next replication to see if that fixes it.
2025-05-22 14212, 2025
Sintharu joined the channel
2025-05-22 14221, 2025
Sintharu
Hi
2025-05-22 14244, 2025
adhawkins[m]
That seems to have done the trick bitmap. Maybe this step could be done automatically whenever a new version of the container is installed?
2025-05-22 14216, 2025
mayhem[m]
No, but outsidecontext might
2025-05-22 14219, 2025
ApeKattQuest joined the channel
2025-05-22 14216, 2025
Maxr1998_ joined the channel
2025-05-22 14248, 2025
Maxr1998 has quit
2025-05-22 14254, 2025
monkey[m]
<Sintharu> "Hi..." <- Hello Sintharu (IRC)
2025-05-22 14259, 2025
monkey[m]
<mamanullah7[m]> "Hey monkey aerozol i needed a..." <- Overall looks good. I don't have much else to say other than what aerozol said: I expect the visual style would follow what we currently do on the connect services page. The LastFM one has extra text inputs for a good example.
2025-05-22 14200, 2025
monkey[m]
And like LastFM, maybe there will be a need for an 'edit' button that is active only when you are connected to these services? Say if you want to change the URL for example.
2025-05-22 14229, 2025
mamanullah7[m]
<monkey[m]> "Overall looks good. I don't have..." <- okay i missed that edit one! i'll make sure to add this!
2025-05-22 14229, 2025
mamanullah7[m]
Thanks monkey aerozol i'll take care suggestions and i'll reach u out for further review!
2025-05-22 14252, 2025
petitminion joined the channel
2025-05-22 14242, 2025
davic has quit
2025-05-22 14243, 2025
spynxic has quit
2025-05-22 14230, 2025
spynxic joined the channel
2025-05-22 14203, 2025
davic joined the channel
2025-05-22 14224, 2025
Sintharu has quit
2025-05-22 14223, 2025
mglubb[m]
Hi yvanzo . Just wanted to say that I'm happy with re-indexing now that I've applied the latest SIR updates and tuned its configuration. Seems to be in the same ballpark as it was before, in terms of time. Possibly a bit quicker. Thank you for your service!
2025-05-22 14224, 2025
davic has quit
2025-05-22 14235, 2025
Sintharu joined the channel
2025-05-22 14216, 2025
Sophist-UK joined the channel
2025-05-22 14235, 2025
davic joined the channel
2025-05-22 14237, 2025
Kladky has quit
2025-05-22 14250, 2025
Kladky joined the channel
2025-05-22 14245, 2025
Sintharu has quit
2025-05-22 14209, 2025
Sintharu joined the channel
2025-05-22 14238, 2025
holycow23[m] joined the channel
2025-05-22 14238, 2025
holycow23[m]
Hey lucifer, I was actually looking into the TimeScale DB to run the a couple of queries over the listens and noticed that it doesn't have the `artist_mbid` or the `recording_mbid` to it, so how exactly do I fetch the same on local for the listens?
2025-05-22 14228, 2025
holycow23[m]
<holycow23[m]> "Hey lucifer, I was actually..." <- Also needed some more assistance with the working of stats, so could we get on a quick zoom call maybe?
2025-05-22 14229, 2025
holycow23[m]
s/more/help/, s/assistance//
2025-05-22 14250, 2025
petitminion has quit
2025-05-22 14246, 2025
petitminion joined the channel
2025-05-22 14201, 2025
yvanzo[m]
Hi mglubb, glad it works for your mirror, sharing your thanks with bitmap and lucifer who made it possible too.
2025-05-22 14250, 2025
yvanzo[m]
lucifer: suggested small changes
2025-05-22 14206, 2025
lucifer[m]
yvanzo: just approved them, thanks!
2025-05-22 14239, 2025
lucifer[m]
<holycow23[m]> "Also needed some more assistance..." <- let's try to work it out over chat first and if it doesn't clear up we can do a call later.
2025-05-22 14203, 2025
holycow23[m]
lucifer[m]: Cool
2025-05-22 14213, 2025
yvanzo[m]
Great, on releasing sir then!
2025-05-22 14218, 2025
lucifer[m]
<holycow23[m]> "Hey lucifer, I was actually..." <- those will come from the mapping data, i'll put up a branch with sample dumps import tomorrow and then you can do a join to `mapping.mb_metadata_cache` table to get artist mbids for listens.
The cron job for the stats how often does it run, I found the file running the cron at /docker/services/cron/crontab, basically I wanna know so if the weekly is chosen then does it run weekly or does it run daily and update the individual time ranges at once?
2025-05-22 14221, 2025
julian45[m]
lucifer: you might already be aware, but just a heads up that the `stable` view of the sir docs (which seems to be the default when browsing to the RTD pages) doesn't yet have the documentation updates you've made, e.g., the [setup page](https://sir.readthedocs.io/en/stable/setup/index.html) still refers to `python2`
2025-05-22 14201, 2025
lucifer[m]
julian45: i am not sure if yvanzo has released the new version yet.
2025-05-22 14212, 2025
yvanzo[m]
On it…
2025-05-22 14243, 2025
lucifer[m]
just checked RTD dashboard, once the release is done stable should update automatically.
the link preview here is probably cached but it has updated to 4.0.1 now.
2025-05-22 14257, 2025
petitminion has quit
2025-05-22 14259, 2025
yvanzo[m]
I disabled link preview, personally.
2025-05-22 14208, 2025
lucifer[m]
ah okay
2025-05-22 14230, 2025
holycow23[m]
* lucifer: The cron job for the stats how often does it run, I found the file running the cron at `/docker/services/cron/crontab`, basically I wanna know so if the weekly is chosen then does it run weekly or does it run daily and update the individual time ranges at once?
2025-05-22 14232, 2025
holycow23[m]
<holycow23[m]> "The cron job for the stats how..." <- also the `get_aggregate_query` is based on the listens table so, over a period of time won't it have all the listens for a period of time or do you filter the listens for a period and then generate the results?
2025-05-22 14250, 2025
outsidecontext[m
<lucifer[m]> "mayhem: do you happen to have..." <- I still do. But I assume you will need OpenSubsonic support (for the MBIDs), as troi requires it?
2025-05-22 14250, 2025
outsidecontext[m
I currently run the release version of funkwhale, but the opensubsonic support is not yet released.
2025-05-22 14224, 2025
lucifer[m]
<holycow23[m]> "also the `get_aggregate_query..." <- can you point me to the query on github?
2025-05-22 14211, 2025
yvanzo[m]
zas: Something went wrong with Solr backup, running it in 3min again.
lucifer[m]: And what is the frequency of the stats update is it done daily?
2025-05-22 14232, 2025
lucifer[m]
yes
2025-05-22 14244, 2025
holycow23[m]
Okay
2025-05-22 14226, 2025
holycow23[m]
Now for example, if I need to do the era stats, it will be based on the release date so that also will be fetched from the dump right?
2025-05-22 14243, 2025
holycow23[m]
But this is the MB Dump right not the Spark dump
2025-05-22 14229, 2025
lucifer[m]
[@holycow23:matrix.org](https://matrix.to/#/@holycow23:matrix.org) not the MB dump but directly from the MB db, we already have that stat for year in music iiuc.
2025-05-22 14225, 2025
lucifer[m]
we have the postgres queries to retrieve the data from the MB db
2025-05-22 14241, 2025
lucifer[m]
in listenbrainz_spark/postgres.
2025-05-22 14202, 2025
holycow23[m]
Yes correct the era one is already in the "Your year in Music 20xx" done but for testing it on local, I could use the MB Dump since I can't use the MB db directly
2025-05-22 14228, 2025
lucifer[m]
That data is cached in hdfs and refreshed daily before stats cron job run.
2025-05-22 14248, 2025
lucifer[m]
The json dump format is different from the database.
2025-05-22 14224, 2025
holycow23[m]
So, for developement of the stats how would I access the DB?
2025-05-22 14254, 2025
lucifer[m]
So it would not work. For this stat, I can export the data and you can import it in your local database.
2025-05-22 14208, 2025
lucifer[m]
For other stats, I can provide you with access to a full mb db replica. You can develop and test your queries on that and then I can export the data for using that query.
2025-05-22 14209, 2025
holycow23[m]
Cool, lemme know how to access the mb db replica
2025-05-22 14243, 2025
lucifer[m]
You can also connect your local spark cluster to a full mb db replica hosted on our servers. Or run a spark cluster on wolf. But that can be slower.
2025-05-22 14233, 2025
holycow23[m]
Could you guide me with connecting the local spark cluster with the full replica?
2025-05-22 14233, 2025
holycow23[m]
Also how do you write queries connecting two different databases, or do you just run individual queries on both?
2025-05-22 14257, 2025
lucifer[m]
The two databases as in?
2025-05-22 14236, 2025
holycow23[m]
Listens would be timescale_db and information regarding the songs would be
2025-05-22 14248, 2025
holycow23[m]
* Listens would be timescale_db and information regarding the songs would be mb_db
2025-05-22 14254, 2025
lucifer[m]
The listens are imported using dumps in spark
2025-05-22 14208, 2025
lucifer[m]
The addition metadata is brought in from MB db
2025-05-22 14219, 2025
lucifer[m]
And then joined together and processed in spark
2025-05-22 14243, 2025
holycow23[m]
So you don't use the timescale_db?
2025-05-22 14257, 2025
lucifer[m]
No that is not used for statistics at all
2025-05-22 14203, 2025
holycow23[m]
aah got it
2025-05-22 14225, 2025
holycow23[m]
got it
2025-05-22 14226, 2025
lucifer[m]
It is only used directly for listens page.
2025-05-22 14257, 2025
lucifer[m]
All the stats, recommendations etc. is done in spark where listen data is imported from dumps.
2025-05-22 14235, 2025
holycow23[m]
Got it, so spark has the listens as well as the info related to recordings, so you run queries over spark and then update the stats
2025-05-22 14243, 2025
lucifer[m]
Yes.
2025-05-22 14211, 2025
holycow23[m]
Thank you so much
2025-05-22 14222, 2025
holycow23[m]
Also till when will the dump be generated?
2025-05-22 14232, 2025
lucifer[m]
The sample data dump?
2025-05-22 14250, 2025
holycow23[m]
The spark dump
2025-05-22 14206, 2025
lucifer[m]
The spark dumps are generated daily for production
2025-05-22 14228, 2025
lucifer[m]
Do you mean the metadata like release dates etc?
2025-05-22 14247, 2025
holycow23[m]
Okay so how exactly would I proceed with my project since I will need the metadata with the listens
2025-05-22 14252, 2025
holycow23[m]
To run queries
2025-05-22 14241, 2025
lucifer[m]
Should be ready early next week.
2025-05-22 14233, 2025
holycow23[m]
Okay, is there anything that I can do before the dump is ready, wanted to start a little early actually
2025-05-22 14237, 2025
lucifer[m]
The data can be exported today but I need to add the import code in spark.
2025-05-22 14251, 2025
holycow23[m]
Okay
2025-05-22 14230, 2025
lucifer[m]
I think you can write the api and frontend side of things meanwhile.
2025-05-22 14248, 2025
lucifer[m]
Using some hardcoded dummy data for testing.
2025-05-22 14206, 2025
holycow23[m]
Okay
2025-05-22 14210, 2025
holycow23[m]
Thanks
2025-05-22 14213, 2025
petitminion joined the channel
2025-05-22 14244, 2025
_BrainzGit
[musicbrainz-server] 14mwiencek opened pull request #3546 (03production…mbs-14032): MBS-14032: Build temporary `release_first_release_date` table for MBS-13966 https://github.com/metabrainz/musicbrainz-server/…
