#musicbrainz-devel

heh.

they have arrived at the wrong project.

hmm.

I wonder if you can I have 1GB sized "objects" in an excel file.

we could open a blank spreadsheet, insert the data dump, save and then mail it off.

that would qualify

do you think their email account would accept a 1gb email anyway?

damn.

hotmail won't. ;-P

this guy seems to be keeping his music metadata in an excel speadsheet. I still can't wrap my head around that.

what did you expect from someone who uses hotmail? :P

it seems that hotmail might be a bit advanced for this guy. :)

it's possible renaming the data dump to '.csv' would be sufficient, actually

excel is probably smarter than this guy

in that *it* knows how to open tab-delimited files :P

but I don't think the data dump would be sufficient for his needs, he probably wants some sort of horrifying multi-joined table :P

select everything, aggregating everything that's potentially multivalues FROM every damn table we have; :P

Wow the forums are really quite orange

happy implement-a-captcha day ;)

hm, I feel like we didn't get all the spam accounts

(that last point there is post-trim)

(as of the dump on the 14th we had 623969 editors)

hm, maybe that's not the peak then? unsure

er, maybe that's not post-trim then

eh, I'll check tomorrow

hrm. not working yet.

So starts another day

<stermi> it stops here

warp: reviewing recaptcha now

cherry1

ffs

wtf. our daily cron job takes over 4 hours to run

am I really reading that correctly?

starts at "Wed Jul 18 00:10:02 UTC 2012", finishes at "Wed Jul 18 04:22:16 UTC 2012"

isn't that including the full dump?

yes, but that is pretty quick

it is? I've noticed my subscriptions email comes late on full dump days

hmm, not quite as slow as i thought

but it still takes an hour before the dump even starts

as fast as i thought*

like today was 05:50, yesterday was 02:52, day before was 02:54, day before that was 02:53...

day before that was 06:43

subscriptions seem to take a while. it's ~25 minutes between the two I get

btw ocharles let me clarify something, I don't disagree with a new NES or the goals of it, i just don't think its the top priority task

and I don't see starting with a database layer that helpful to creating NES either, the input/ouput of NES is going to be the webservice (i.e submit edit, view edit) so it makes more sense to me to

consider how to implement these tasks and break them down, rather than created some database calls that amy not ultimately be what is required

Which is one of may reasons why I think the most useful thing we could do is separate the website from the web service AND get website to use the web service for all database access

yes, but that's a very large amount of work to take in one go

i don't think we can coordinate that much work at once

so what I want to do is keep the website the same, but change how the Data::* section works

then once that is done, and the website is talking to a NES backed database, we can change the website to start talking to our public web service

I don't plan to start writing database methods that won't be called, i agree - that's fairly pointless

The 'very large amount of work' argument is a different one, its not your main drive though is it. i.e if we had masses of developers which way would you approach it ?

probably still the same way

but that seems irrelevant, we don't have that, and we're not going to, so why choose a strategy that doesn't apply to us?

I'm interested in talking small progressive steps to nirvana, not rewriting everything and experiencing NGS again

Well, because there are easy ways round that. You could create a new web service layer from scratch, and just code the NES stuff from web service layer down to db as stage 1

stage2:website talks to new webservice

stage 3:merge new web service with old

that seems to require that we have a design for /ws/3 as well, no?

Of course you don't plan to start writing database methods that won't be called, but it seems very difficult to avoid working bottom up

No, i dont see that has to be done upfront

how can we write a web service if we don't know what the web service is?

How is this different to the website talking to a NES backed database

Im saying you don't need to define a whole ws/3 upfront, but clearly you need to think about how you want the website to interact with the database/webservice either way

the website is already defined in terms of some general operations, so we're just reimplementing them with a NES database, rather than the NGS database

I don't like the idea of writing an ultimately public facing web service without a clear design first. building it adhoc in terms of what we need seems like a recipe for yet another poorly scaling WS

The idea of writing a new data layer that the website talks to without considering the web service that the website is meant to talk to doesn't sound like an efficient use of resources to me

re ws/2 scalability it would be good to have this documented somewhere, other than the problems with a few artists called with a particular set of inc arguments I'm not aware of all what all the other problems are

ijabz: the fact that it's near impossible to set Last-Modified

that's more my bigger worry

right, can see that issue, would be interested in what the alternative solution is you have in mind, i know you want to get rid of incs, but not sure what the replacement is you have in mind

nor am I, because I figured I'd focus on building NES first :)

i personally think even trying to do the editable web service at the time is a bad idea and we're biting off more than we can chew with even just NES

at the same time*

I thought editable web service was on hold until NES had progressed

warp's work on JSON for /ws/2 is part of editable ws

and we seem to have lost warp so I better adopt this recaptcha work

Well so far there is nothing editable about it, it long been requested to provide ws output as json so I see no conflict

there is value in providing json

but the motivation is because we will only support writing via json, thus we want read/write symmetry

But that is irrelevant, the read still need to be done, and is not causing aproblem

in fact the NES roadmap you've laid out seems to create an awful lot of bottlenecks , so no other major work can be done until you've finished

i.e editable web service, separate website from web service, move way from perl

yes, i see personally see NES as more important than any of those

editable WS is very close though

but i don't see value in an editable WS that isn't on NES, when we plan to have NES so soon. introducing an api then almost immediately deprecating it is not very social

taking your small progressive steps analogy, working now on editable ws based on current system, and then updating it to support the new features of new seems reasonable to me

the editable web service doesn't have to actually be released and made public until works with NES

what's the point in writing it until NES then?

warp: ok, my suggestion for HTML::FHX::Field::reCAPTCHA is not working out. throwing that work away and shipping yours

small steps, parallel working, the lapsed time to delivery is going to be much longer if work isn't started until you've finished NES

my small steps is about small *deliverable* steps

and if you can't deliver something until something else is done, you might as well wait for it to be done. sure you could do 'the rest' while waiting, but there isn't really anything that depends on NES

plus, it's entirely possible i haven't got NES right and it has to be refined

'there isn't really anything that depends on NES' now Im confused, whats the problem then ?

warp, you haven't commited register.js :(

anything else*

i meant there is hardly anything to write about the editable web service without NES

the entire service builds on it

I don't see that, we have all the web service side for a start

Anyway, can't spend all day chatting of for some food

hello!

ocharles: the recaptcha branch is supposed to remove register.js

warp: it didn't, but it does now

i'm just about to deploy it

cool

did you replace the REMOTE_ADDR stuff with $c->req->address ?

yea

it seems to work on test

good :)

cool, I got a google streetview building/house number in the captcha

lol

ocharles: tested entering an incorrect and correct captcha, both seem to work as expected on test.musicbrainz.org.

warp: do we have musicbrainz.org keys?

I made some this morning, yes.

those are the ones in DBDefs on test.

ok

but they're tied to a private gmail account. so they'll work for now, but we need to move them somewhere else.

(which probably means generating new keys)

hum. no captcha on asterix still

oh, it's not using $c->req->address cause i forgot to push

hah

nope, still nothing

presumably you configured DBDefs on asterix correctly