Do not take it for granted that standards wrt barcodes will be stringently followed by everyone
2025-09-23 26605, 2025
LupinIII has quit
2025-09-23 26623, 2025
LupinIII joined the channel
2025-09-23 26640, 2025
LupinIII has quit
2025-09-23 26658, 2025
LupinIII joined the channel
2025-09-23 26625, 2025
isobel_86 has quit
2025-09-23 26609, 2025
tagomago joined the channel
2025-09-23 26632, 2025
Maxr1998_ joined the channel
2025-09-23 26648, 2025
Maxr1998 has quit
2025-09-23 26620, 2025
zer0bitz-_ joined the channel
2025-09-23 26613, 2025
zer0bitz- has quit
2025-09-23 26603, 2025
G0d joined the channel
2025-09-23 26643, 2025
MeatPupp3t21 has quit
2025-09-23 26611, 2025
MeatPupp3t21 joined the channel
2025-09-23 26642, 2025
MeatPupp3t21 has quit
2025-09-23 26606, 2025
MeatPupp3t21 joined the channel
2025-09-23 26628, 2025
zer0bitz-_ has quit
2025-09-23 26637, 2025
zer0bitz- joined the channel
2025-09-23 26659, 2025
innocuou- joined the channel
2025-09-23 26602, 2025
innocuous has quit
2025-09-23 26634, 2025
Isobel_86 joined the channel
2025-09-23 26609, 2025
v6lur_ joined the channel
2025-09-23 26659, 2025
MeatPupp3t21 has quit
2025-09-23 26625, 2025
MeatPupp3t21 joined the channel
2025-09-23 26654, 2025
PaulDOkno[m] joined the channel
2025-09-23 26655, 2025
PaulDOkno[m]
Hello, there. Is it the right place to ask technical question about the musicbrainz database ?
2025-09-23 26615, 2025
MeatPupp3t21 has quit
2025-09-23 26641, 2025
MeatPupp3t21 joined the channel
2025-09-23 26608, 2025
PaulDOkno[m]
btw Hello yvanzo (IRC) it's been a while :)
2025-09-23 26649, 2025
reosarevok[m]
This or #metabrainz:chatbrainz.org anyway. Potentially that one is best, more devs there
2025-09-23 26606, 2025
PaulDOkno[m]
Okay, I will go ahead here then. I am writting a Lidarr metadata server and I am currently initializing a local musicbrainz database (the process looks a lot like what mbslave is doing). My issue is with the Live Data Feed ingestion. Ingesting replication packet takes a while. I suspect this is because I have added some triggers of my own to keep a Meilisearch index in sync with the database. I guess the default replication
2025-09-23 26606, 2025
PaulDOkno[m]
process for musicbrainz mirror does not have this issue, but I was not able to determine how it was done. Maybe there are some existing sql triggers on mirrors that I missed ?
2025-09-23 26609, 2025
PaulDOkno[m]
Long story short these triggers https://github.com/oknozor/metadada/blob/feat/mb-… makes sql transaction time skyrocket during live datafeed ingestion. I think I am missing the default replication process for miirrors.
2025-09-23 26640, 2025
MeatPupp3t21 has quit
2025-09-23 26604, 2025
MeatPupp3t21 joined the channel
2025-09-23 26614, 2025
MeatPupp3t21 has quit
2025-09-23 26640, 2025
MeatPupp3t21 joined the channel
2025-09-23 26607, 2025
v6lur_ has quit
2025-09-23 26641, 2025
yvanzo[m] joined the channel
2025-09-23 26642, 2025
yvanzo[m]
Hi Paul D. Okno, it’s been a while indeed, welcome back 🧙... (full message at <https://matrix.chatbrainz.org/_matrix/media/v3/download/chatbrainz.org/OxxaTMzIooqaUaxNRJbkKxGm>)
2025-09-23 26653, 2025
PaulDOkno[m]
Actually after doing a bit of digging into postgres internal, it's not my custom indexing trigger that are taking time. But tag ref_count updates. I don't know if those comes from replication packet or are triggered automatically on update.
2025-09-23 26658, 2025
PaulDOkno[m] uploaded an image: (180KiB) < https://matrix.chatbrainz.org/_matrix/media/v3/download/hoohoot.org/YRjLCYwntZVTgzgNFdMmspbG/image.png >
2025-09-23 26651, 2025
PaulDOkno[m]
Packet takes between 1 to 5 minutes to get ingested. Maybe I shouldn't be worried. How does that compare to a typical mirror setup ?
2025-09-23 26618, 2025
PaulDOkno[m]
Also while developing this I wrote a mbslave clone but with fancy progress bar, the whole musicbrainz database setup takes about 20~40min (on just a subset of the database). For now its included in a bigger project but maybe it would be worth to turn this into a standalone binary for other people to use.... (full message at <https://matrix.chatbrainz.org/_matrix/media/v3/download/chatbrainz.org/ajqQQsQFWchxFPBPiqVxZISU>)
2025-09-23 26654, 2025
vzctr81191 joined the channel
2025-09-23 26601, 2025
joessexmpp joined the channel
2025-09-23 26627, 2025
joessexmpp has quit
2025-09-23 26644, 2025
lusciouslover has quit
2025-09-23 26621, 2025
lusciouslover joined the channel
2025-09-23 26639, 2025
reosarevok[m]
Each hourly packet taking at most 5 minutes doesn't seem like the worst at least :)
2025-09-23 26647, 2025
reosarevok[m]
mbslave but with automatic schema update would probably make some people happy seeing the current issues in mbslave with people's dbs failing because of schema changes
2025-09-23 26608, 2025
LupinIII has quit
2025-09-23 26626, 2025
LupinIII joined the channel
2025-09-23 26643, 2025
LupinIII has quit
2025-09-23 26601, 2025
LupinIII joined the channel
2025-09-23 26643, 2025
Cheezmo_ joined the channel
2025-09-23 26657, 2025
Cheezmo_ has quit
2025-09-23 26618, 2025
krei-se has quit
2025-09-23 26611, 2025
lusciouslover has quit
2025-09-23 26607, 2025
lusciouslover joined the channel
2025-09-23 26612, 2025
PaulDOkno[m]
Actually my first estimation was not accurate, some packet take between 1 to 5 mins some other can take as long as 1h30 :(
2025-09-23 26620, 2025
krei-se joined the channel
2025-09-23 26613, 2025
PaulDOkno[m]
so I think I am missing something, currently I am executing all pending data in a single sql transaction. Also streaming directly from the tar archive (no write to dbmirror2.pending_data/pending_key). I am just updating current_replication_sequence and current_replication_schema after each packet have been applied
2025-09-23 26612, 2025
PaulDOkno[m]
Does the official replication process involve grouping by xid and removing pending data from db mirror to keep track of the remains in case of failure ?
2025-09-23 26608, 2025
Isobel_86 has quit
2025-09-23 26635, 2025
k-man has quit
2025-09-23 26600, 2025
k-man joined the channel
2025-09-23 26623, 2025
v6lur_ joined the channel
2025-09-23 26633, 2025
monkey[m] has quit
2025-09-23 26607, 2025
Island joined the channel
2025-09-23 26647, 2025
tmalloc has quit
2025-09-23 26640, 2025
tmalloc joined the channel
2025-09-23 26632, 2025
lusciouslover has quit
2025-09-23 26648, 2025
lusciouslover joined the channel
2025-09-23 26619, 2025
walkfly joined the channel
2025-09-23 26642, 2025
antlarr
I think https://musicbrainz.org/release/1e0e1c01-a63c-403… is wrong and the release (and release group) artist should be "Various Artists" since it's just a compilation of individual recordings and not an album those artists recorded together. But before doing that change I'd like to know others' opinions
2025-09-23 26657, 2025
antlarr
actually... ignore that
2025-09-23 26653, 2025
antlarr
it's really a rerecording made by Christopher Warren‐Green and the London Chamber Orchestra of those songs
2025-09-23 26631, 2025
antlarr
s/a rerecording/covers/
2025-09-23 26650, 2025
antlarr
although then... I'm not sure it should be marked as "compilation"
2025-09-23 26621, 2025
worldlooksblack joined the channel
2025-09-23 26638, 2025
yvanzo[m]
Paul D. Okno: I don’t know, but dbmirror2’s author will be back from holiday next week.
[@yvanzo:chatbrainz.org](https://matrix.to/#/@yvanzo:chatbrainz.org) thanks, but I need just a subset of the database, roughly artists+release+release_group.So just like mbslave does, I filter data from the dump and from the replication feed on configured tables. Also the indexation is done with meilisearch, which is really lightweight compared to solr, allowing to run the app on really cheap hardware (my homelab)
2025-09-23 26606, 2025
PaulDOkno[m]
But I definitely have some question for the dbmirror2 dev :)
2025-09-23 26655, 2025
worldlooksblack has quit
2025-09-23 26613, 2025
worldlooksblack joined the channel
2025-09-23 26601, 2025
yvanzo[m]
I see. A selective-replication mechanism had been thought about but it isn’t planned. Selective live Solr indexing is optionally available but it is definitely heavy-weight.
2025-09-23 26648, 2025
worldlooksblack has quit
2025-09-23 26648, 2025
worldlooksblack joined the channel
2025-09-23 26637, 2025
reosarevok[m]
antlarr (IRC): it's a compilation if the recordings themselves were already released before separately
2025-09-23 26647, 2025
reosarevok[m]
If not, removing compilation is right
2025-09-23 26652, 2025
rustynova[m] joined the channel
2025-09-23 26652, 2025
rustynova[m]
~~Don't count on me to come maintain that one lol~~
