only potential issue here is that replication packets use row ids
2018-03-15 07431, 2018
alastairp
so we'd have to set those specifically too (not a huge problem)
2018-03-15 07407, 2018
alastairp
rsh7: are you following along?
2018-03-15 07424, 2018
alastairp
replication packets are the way that mirrors of the musicbrainz database keep up to date
2018-03-15 07436, 2018
alastairp
it's effectively a file that says "change this row in this table to this value"
2018-03-15 07455, 2018
alastairp
iliekcomputers is suggesting that if we keep the table structure exactly the same, we can just look in the replication packet file and see if the item being updated exists in our mirror, and if so, just copy it in
2018-03-15 07411, 2018
alastairp
this means that the musicbrainz update system takes care of the "last_updated" field for us
2018-03-15 07458, 2018
alastairp
ruaok: bitmap: confirm if you think this sounds like a reasonable idea?
2018-03-15 07426, 2018
iliekcomputers
me and rsh7 talked about this a bit, but we weren't sure how easy / hard it would be to implement
2018-03-15 07449, 2018
alastairp
remember: the reason that we want a copy of some musicbrainz metadata in the ab database is to be able to join against it in a single query to do filtering/stats
2018-03-15 07411, 2018
alastairp
because a second database would require us to do queries to two databases and join in python
2018-03-15 07442, 2018
iliekcomputers
like the problem of showing only those release-groups that had a review in CB ferbncode was trying to solve
2018-03-15 07407, 2018
alastairp
the only potential problem is that the number of tables we have to copy becomes so big that it's as unweildy to deal with as querying two databases
alastairp, iliekcomputers: One problem with replication is that the db will always be behind, and there *will* be users that add something in MB and expect to use/see it in *B straight away.
2018-03-15 07406, 2018
Freso
(This has been an issue for AB previously too, and continues to be an issue with Picard/AcoustID interaction.)
2018-03-15 07454, 2018
alastairp
Freso: it's possible that we could special-case this: if an mbid exists in AB and is not in our metadata tables, do a ws lookup, if that returns data then show it along with an infobox
2018-03-15 07421, 2018
bitmap
alastairp: if the schema is a superset of mb's, you could probably use the scripts mb provides to just apply the packets? I think they might skip tables that don't exist, though if not it would be trivial to add an option allowing that
2018-03-15 07423, 2018
sud0pl0x joined the channel
2018-03-15 07449, 2018
samj1912
bitmap, can I bug you with MB PR reviews? :p
2018-03-15 07400, 2018
bitmap
you may
2018-03-15 07429, 2018
samj1912
Election pages are done from my side
2018-03-15 07449, 2018
samj1912
Ditto with the stash move
2018-03-15 07403, 2018
alastairp
bitmap: right, the only other potential issue is that I don't want to store all MB rows in our local tables
2018-03-15 07442, 2018
alastairp
e.g., we only have 3M unique recordings in AB, no reason to store 18M recording rows
2018-03-15 07407, 2018
bitmap
it definitely already skips UPDATEs and DELETEs for rows that don't exist, but I suppose you want to filter out inserts too
2018-03-15 07432, 2018
alastairp
right
2018-03-15 07408, 2018
alastairp
(I have no idea what timezone you indian crowd are on)
2018-03-15 07421, 2018
alastairp
if it was 2am here I'd be asleep a long time ago
2018-03-15 07454, 2018
rsh7
it's 1:22 am 😅
2018-03-15 07405, 2018
ferbncode[m]1
:D
2018-03-15 07417, 2018
bitmap
all inserts, or do you want to apply a test to each row to determine if it should be added? (sorry I didn't catch what the exact use case was)
2018-03-15 07419, 2018
iliekcomputers
late sleepers
2018-03-15 07421, 2018
alastairp
all inserts, I think
2018-03-15 07428, 2018
iliekcomputers
so a good way to update the copied data would be:
2018-03-15 07431, 2018
alastairp
we only want to insert a row when that mbid is added to acousticbrainz
2018-03-15 07450, 2018
alastairp
which I think we'd do with our own script that runs against an existing MBS slave db
2018-03-15 07406, 2018
iliekcomputers
1. download the replication packets on intervals
2018-03-15 07426, 2018
iliekcomputers
2. just apply them (without much modification because it skips rows that don't exist already?)
2018-03-15 07435, 2018
iliekcomputers
seems simple enough?
2018-03-15 07450, 2018
iliekcomputers
or am i missing something?
2018-03-15 07422, 2018
bitmap
we could add a couple options to LoadReplicationChanges to skip tables that don't exist, and skip inserts
2018-03-15 07423, 2018
alastairp
rsh7: it sounds like the tools that we have available are going to let us do it this way. update your proposal to cover the results of this discussion
2018-03-15 07432, 2018
alastairp
the "difficult" step will be to decide which tables to replicate
2018-03-15 07440, 2018
alastairp
see if you can come up with a list
2018-03-15 07457, 2018
alastairp
iliekcomputers: do you know your way around the mb database?
2018-03-15 07420, 2018
iliekcomputers
alastairp: not very well
2018-03-15 07429, 2018
alastairp
great, you can help :)
2018-03-15 07439, 2018
iliekcomputers
I mean, I have a basic idea, but I wouldn't be pop quiz ready
2018-03-15 07442, 2018
alastairp
and on that note, given that it's 9pm, I'm going to bed
2018-03-15 07449, 2018
alastairp
have a good night/morning all
2018-03-15 07451, 2018
iliekcomputers
(i don't know what l_url_url does)
2018-03-15 07456, 2018
ferbncode[m]1
alastairp: there will be a slave MB db connected to AB (or we can use WS too) and we update tables in AB whenever a new recording is added (inserts case). Right?
2018-03-15 07407, 2018
alastairp
yes
2018-03-15 07413, 2018
iliekcomputers
alastairp: good night 😴
2018-03-15 07415, 2018
ferbncode[m]1
It's empty :D
2018-03-15 07441, 2018
iliekcomputers
ferbncode[m]1: the idea is to use a similar connection to the one in CB to insert data into the AB database
adhawkins: I reverted the commit that caused LCA-7, so it can be closed now (unless you still want to fix the code to carry over the query params, just to future-proof things)
