ocharles: what is the difference between views and whatever you suggested previously, and does the difference matter for the rest of this discussion?
2013-02-20 05155, 2013
ruaok
1. you must not understand NES to query our data.
2013-02-20 05111, 2013
ruaok
2. NES must not slow data fetching down more than 10% over our current system
2013-02-20 05124, 2013
ruaok
and clearly if its 12% then thats not a deal breaker.
2013-02-20 05130, 2013
ocharles
Ok
2013-02-20 05100, 2013
ruaok
thus any proposed solutions you put forth need to be evaluated from performance perspective.
2013-02-20 05101, 2013
ocharles
If 1 must hold forever, then I think we are proposing that we should offer a schema/database dumps that have no historical information
2013-02-20 05105, 2013
ocharles
which is pre-NES right now
2013-02-20 05123, 2013
ruaok
yep, I'm totally fine with that.
2013-02-20 05134, 2013
nikki
I thought we wanted that anyway?
2013-02-20 05138, 2013
ruaok
so far, not a single customer has given the slightest fuck about historical data.
2013-02-20 05143, 2013
ocharles
Ok, that does mean supporting two database schemas - as the NES (or historic) one changes, so will the other one have to
2013-02-20 05150, 2013
ocharles
ok
2013-02-20 05117, 2013
ocharles
i'm just trying to understand the the discarding of time information is not something that should be a hack, but must be a real shippable product
2013-02-20 05121, 2013
ruaok
did I answer your question? if not, can you please restate it?
2013-02-20 05127, 2013
ocharles
let me ramble a sec longer :)
2013-02-20 05132, 2013
ruaok listens
2013-02-20 05156, 2013
warp
ocharles: of the fullexports, they're commonly imported without edits. it wouldn't suprise me if most of our clients are similarly not interested in the history.
2013-02-20 05117, 2013
ocharles
so, we're agreed we need NES (obviously), but also a view on NES that discards everything but the 'official' version of entities. This database should have replication.
2013-02-20 05132, 2013
ruaok
+1
2013-02-20 05142, 2013
ocharles
because this needs to be done well, and exist forever, it should not be a hack
2013-02-20 05151, 2013
warp
do we need replication for the version with history? or are dumps sufficient?
2013-02-20 05152, 2013
ocharles
I want more people onboard with NES, and this is exactly the kind of thing warp or ian could work on
2013-02-20 05100, 2013
ruaok
the history of changes is only relevant inside the MB universe. not outside it.
2013-02-20 05112, 2013
ruaok
warp: no history in replication.
2013-02-20 05118, 2013
ocharles
it gives them a way to really understand NES, and is entirely parallel to me
2013-02-20 05124, 2013
ruaok
there was one company who has asked for edit data to be replicated.
2013-02-20 05134, 2013
ruaok
but they realized they were asking for the wrong thing. :)
2013-02-20 05134, 2013
ocharles
I think we want replication for history as well, but if it's a separate database entirely, it's a separate replication stream
2013-02-20 05143, 2013
ocharles
people here have wanted edit replication
2013-02-20 05149, 2013
ocharles
at least, that was my understanding
2013-02-20 05158, 2013
warp
ruaok: our end-users are certainly interested in history, if they are included in your definition of "MB universe", your statement is correct. otherwise not :)
2013-02-20 05100, 2013
ianmcorvidae
I think supporting replication for the full stream is incredibly easy, since *that* is just replicating like we already do
2013-02-20 05100, 2013
ocharles
but that's another topic. the replication we *need* is non-historic
2013-02-20 05115, 2013
ianmcorvidae
the non-historic is the interesting one though, yes :)
2013-02-20 05127, 2013
warp
ruaok: haha
2013-02-20 05129, 2013
ocharles
so warp, ianmcorvidae - how does the suggestion of writing this mapper sound?
2013-02-20 05135, 2013
ruaok
warp: yes, end users are MB universe.
2013-02-20 05152, 2013
ruaok
I use the term customers to describe downstream data users.
2013-02-20 05105, 2013
ruaok
those who just want the data, but dont want to deal with the unwashed metadata hippies.
2013-02-20 05120, 2013
ianmcorvidae
I'm never sure what to say to that sort of question, ocharles; I'm interested but spread very thin (as are we all, I think -- but still)
2013-02-20 05138, 2013
warp
I think I'm booked solid and then some until the schema change.
2013-02-20 05155, 2013
ocharles
ok
2013-02-20 05152, 2013
ruaok
I'd say lets get the schema change coding out the way and then have a meeting to divvy up NES work.
2013-02-20 05104, 2013
ruaok
but I think warp is right that we're all going to be busy with schema change stuff.
2013-02-20 05116, 2013
ocharles
so far I'm at the decisions: we need a mapping service, and search should consume the database provided by this
2013-02-20 05128, 2013
ruaok
and, now that we spend 3 months on a schema change, we're spending 50% of out lives in schema change mode. :(
2013-02-20 05129, 2013
ocharles
so /ws is the only remaining point of interest
2013-02-20 05143, 2013
ruaok
oh, I learned that saying "schema change" is verboten at discogs. plain tabu. :)
2013-02-20 05149, 2013
warp
ruaok: how long do we need to keep /ws/1 running?
2013-02-20 05108, 2013
ianmcorvidae
ruaok: you'll have to explain that further some time :)
2013-02-20 05109, 2013
ruaok
warp: judging by history at least another 4 years. :(
2013-02-20 05113, 2013
ruaok
ianmcorvidae: k
2013-02-20 05115, 2013
ocharles
I think for that, I should work out excatly how much work has to be done to move that to NES -- if there's a big intersection of the website and /ws/s, then I think we should write the /ws talking to NES for now
2013-02-20 05123, 2013
warp
ruaok: f*ck.
2013-02-20 05144, 2013
ocharles
Moving /ws to talk to a separate database from the main site for most things, except writes, feels like a massive pita
2013-02-20 05102, 2013
ocharles
with search, that separation is almost already done
2013-02-20 05122, 2013
warp
I was hoping we could just fork /ws/1, run the current codebase against the mapped database on a seperate dodgy frontend and then forget about it.
2013-02-20 05123, 2013
ruaok
I think we could probably get away with 1 - 2 years, but you know how lazy our WS consumers are. :(
2013-02-20 05146, 2013
warp
but if we need to keep that running for another four years, /ws/1 needs to stay part of the codebase we maintain :/
2013-02-20 05125, 2013
ianmcorvidae
I propose we start figuring out who's still using it and start hounding them :P
2013-02-20 05130, 2013
ruaok
warp: sadly I need no way for us to ditch the /ws/1 by the time NES comes around.
2013-02-20 05146, 2013
warp
ruaok: not by the time NES comes around
2013-02-20 05147, 2013
ruaok
ianmcorvidae: I'd like that, but I think that our consumers are already kinda pissed at us.
2013-02-20 05114, 2013
ruaok
and if you've paid attention to the comments on the proposed /ws/2 changes, people want NOTHING to change.
2013-02-20 05125, 2013
ruaok
they are not even willing to entertain the actual changes.
2013-02-20 05135, 2013
warp
ruaok: but it doesn't need any changes really, we cannot implement new features in /ws/1/ anyway. so if there is a database which looks like the current schema for it to talk to, we will not have to touch the code.
2013-02-20 05140, 2013
ruaok
one person suggested that we come up with a v2 of the web service.
2013-02-20 05150, 2013
ruaok
not even cluing in that were were talking about v2.
2013-02-20 05153, 2013
ocharles
warp: ws/1 still has write methods in
2013-02-20 05159, 2013
ruaok shakes his head and deposits 10p
2013-02-20 05110, 2013
warp
ocharles: oh, meh.
2013-02-20 05126, 2013
ruaok
but we agree that the write methods are minor.
2013-02-20 05133, 2013
Freso
We could retire ws/1's write methods and require that new writes go through ws/2?
2013-02-20 05138, 2013
ruaok
ratings, tags, discids, smurfs, right?
2013-02-20 05140, 2013
ocharles
we could indeed do that
2013-02-20 05144, 2013
Freso
That might also speed up adoption of /2...
2013-02-20 05147, 2013
ruaok
Freso: good one.
2013-02-20 05148, 2013
ruaok
I like that.
2013-02-20 05107, 2013
Freso is contributing to NES o/
2013-02-20 05127, 2013
bandtrace joined the channel
2013-02-20 05133, 2013
nikki
ruaok: ratings, tags, isrcs and collections, isn't it?
2013-02-20 05138, 2013
warp
anyway, whether /ws/1 is seperate or not is not that a terribly interesting issue on its own. I just like the idea of pruning our mbserver code from irrelevant bits.
2013-02-20 05139, 2013
nikki
and puids
2013-02-20 05146, 2013
ruaok
collections == smurfs, right? :)
2013-02-20 05152, 2013
hawke_1
can we drop puids yet? :-p
2013-02-20 05157, 2013
Freso
Yeah, pweeds too.
2013-02-20 05159, 2013
Freso
hawke_1: No.
2013-02-20 05100, 2013
ruaok
hawke: that would solve that too.
2013-02-20 05105, 2013
Freso
hawke_1: Not until October.
2013-02-20 05116, 2013
Freso
hawke_1: We decided that ~3 weeks ago at a dev. meeting.
2013-02-20 05119, 2013
hawke_1
k
2013-02-20 05125, 2013
ruaok
and NES isn't going to happen by october, so that suggestion is valid, i think.
2013-02-20 05126, 2013
ocharles
i doubt nes will be done by october
2013-02-20 05128, 2013
Freso wrote the notes and read them over several times :p
2013-02-20 05145, 2013
Freso
Well, we'll see by then, won't we? :)
2013-02-20 05103, 2013
ruaok
I saw how long NGS took.
2013-02-20 05113, 2013
Freso
If pweeds are wiped by October and NES isn't out, we don't need to worry about them.
2013-02-20 05113, 2013
ruaok
ocharles is amazing, but still, it aint going to be done by then.
2013-02-20 05125, 2013
ruaok
Freso: yep.
2013-02-20 05143, 2013
Freso
If NES is *almost ready* by October, we can probably do them in that much earlier (or just drop the ws write support...).
2013-02-20 05157, 2013
warp
let's get back ontopic.
2013-02-20 05101, 2013
Freso
Or rather, if NES is ready and it's *almost Oct...
2013-02-20 05102, 2013
ruaok
lets summarize.
2013-02-20 05129, 2013
ruaok
as for the /ws/1, we will stop allowing writes at NES release.
2013-02-20 05152, 2013
ruaok
and PUID support will also be dropped because PUIDs will be gone before NES is released.
2013-02-20 05102, 2013
ruaok
right?
2013-02-20 05110, 2013
ocharles
good so far
2013-02-20 05115, 2013
ruaok
onward!
2013-02-20 05158, 2013
warp
we need to string /ws/1 along for the time being.
2013-02-20 05114, 2013
ruaok nods
2013-02-20 05117, 2013
ocharles
but that can be done by using the current /ws/1 in musicbrainz-server and running it separately
2013-02-20 05122, 2013
ocharles
(pointed to a mapped database)
2013-02-20 05131, 2013
ruaok is a bobblehead
2013-02-20 05136, 2013
warp
the musicbrainz.org website will get changes to its Data:: layer to talk directly to the real NES
2013-02-20 05101, 2013
warp
/ws/1 and /ws/2 still somewhat undecided if they talk to NES or to mapped views.
2013-02-20 05112, 2013
ruaok
correct.
2013-02-20 05115, 2013
ocharles
i think /ws/1 is a nobrainer now
2013-02-20 05119, 2013
ruaok
that depends on the performance.
2013-02-20 05126, 2013
ruaok bobbles moer
2013-02-20 05130, 2013
ruaok
moar!
2013-02-20 05104, 2013
warp
ocharles: if we need to string /ws/1 along for four years, it should just do the same thing as /ws/2 does.
2013-02-20 05120, 2013
ocharles
but it's read-only
2013-02-20 05120, 2013
ruaok takes a quick loo break
2013-02-20 05122, 2013
ocharles
and non-historic
2013-02-20 05123, 2013
ocharles
but ok
2013-02-20 05125, 2013
bandtrace joined the channel
2013-02-20 05152, 2013
warp
ocharles: in four years we will have schema changes. which will required backporting in some fashion to keep /ws/1 running.
2013-02-20 05107, 2013
ocharles
they already require that to keep the non-historic db running
2013-02-20 05114, 2013
ocharles
but ok, point taken
2013-02-20 05120, 2013
ocharles
that might be breaking enough that /ws/1 has to have code changes
2013-02-20 05122, 2013
warp
ocharles: either we do that backporting in the /ws/1 perl code, or it may need a seperate (specific to /ws/1) mapped NES
2013-02-20 05103, 2013
warp
ocharles: I'm talking about things like how we made multi-disc releases work (to some extent) in current /ws/1, and changes like that.
2013-02-20 05110, 2013
ocharles
yea
2013-02-20 05130, 2013
ruaok returns
2013-02-20 05134, 2013
warp
ocharles: would it be feasible to do mappings like that with database views?
2013-02-20 05142, 2013
warp
ocharles: even if the views would be specific to /ws/1?
2013-02-20 05155, 2013
warp
ocharles: or would it better to adapt the /ws/1 perl code?
2013-02-20 05159, 2013
ocharles
not sure i follow
2013-02-20 05124, 2013
ocharles
but if you mean future schema changes, it really depends on the nature of the change - so we shouldn't discuss that in a meeting summary :)
2013-02-20 05150, 2013
warp
ocharles: well, that's the problem, we will not know the nature of the changes we will be making three years from now.
2013-02-20 05101, 2013
ocharles
no, so we deal with them when they happen
2013-02-20 05122, 2013
warp
ocharles: so which of the two approaches is more likely to be able to cope with such changes, and will take the least amount of resources to adapt to the changes?
2013-02-20 05104, 2013
ocharles
i think they are equal. if /ws/1 talks to NES and nes changes, /ws/1 must change. if /ws/1 talks to a mapping, and that changes, then /ws/1 must change
2013-02-20 05113, 2013
ocharles
I guess the mapping is slightly less likely to change than NES, but still
2013-02-20 05101, 2013
warp
if /ws/1 talks to a mapping and NES changes, you adapt the mapping. but you're fucked if the mapping cannot ... er... do the mapping :)
2013-02-20 05103, 2013
ianmcorvidae
the mapping seems like it'd change just as much as the NES schema -- they're supposed to be parallel, and other than history should be at parity, no?
