0:15 AM
kurtjx joined the channel
2012-11-01 30600, 2012
0:27 AM
kepstin joined the channel
2012-11-01 30651, 2012
0:29 AM
ocharles
adhawkins-away: sounds like you're making progress!
2012-11-01 30651, 2012
0:33 AM
reoafk joined the channel
2012-11-01 30643, 2012
0:53 AM
kurtjx joined the channel
2012-11-01 30600, 2012
1:07 AM
reoafk joined the channel
2012-11-01 30638, 2012
1:20 AM
Freso joined the channel
2012-11-01 30639, 2012
1:39 AM
ianmcorvidae attempts to install elasticsearch
2012-11-01 30601, 2012
2:06 AM
Prophet5 joined the channel
2012-11-01 30618, 2012
2:06 AM
ianmcorvidae
cool, got ingestr running
2012-11-01 30626, 2012
2:06 AM
ianmcorvidae
I mean, it doesn't actually *do* anything since I have no data, but :P
2012-11-01 30640, 2012
2:06 AM
nikki
get some data then!
2012-11-01 30634, 2012
2:07 AM
reosarevok
You have the IA data!
2012-11-01 30637, 2012
2:07 AM
reosarevok
Use it :p
2012-11-01 30631, 2012
2:08 AM
ianmcorvidae
at present it really only supports some datasets I don't have :P but yes, that'll be my next step
2012-11-01 30610, 2012
2:11 AM
reosarevok
ianmcorvidae: while you're at it, it could probably use some option to add releases in low quality by default :p
2012-11-01 30612, 2012
2:12 AM
ianmcorvidae
initially, it won't support adding releases at all :P it needs to get data into its own DB before it can worry about getting it into MB's
2012-11-01 30606, 2012
2:14 AM
reosarevok
Ooh
2012-11-01 30607, 2012
2:14 AM
reosarevok
cool
2012-11-01 30611, 2012
2:14 AM
reosarevok
Import the Naxos Music Library :p
2012-11-01 30602, 2012
2:15 AM
reosarevok
Or the BIS site, or Chandos or Hyperion or or or
2012-11-01 30611, 2012
2:15 AM
reosarevok
So many things I'd love to see ingested...
2012-11-01 30628, 2012
2:15 AM
reosarevok goes ingest some chips for now
2012-11-01 30600, 2012
2:18 AM
ianmcorvidae
I think I'll start with the IA data :P
2012-11-01 30636, 2012
2:18 AM
reosarevok
Sure, sure
2012-11-01 30641, 2012
2:18 AM
reosarevok
Ingest it all!
2012-11-01 30656, 2012
2:18 AM
reosarevok
does it have any options to find dupes?
2012-11-01 30600, 2012
2:19 AM
reosarevok
(from multiple datasets)
2012-11-01 30634, 2012
2:19 AM
reosarevok
I guess it doesn't need them yet, but it will at some point
2012-11-01 30645, 2012
2:20 AM
ianmcorvidae
that'll be your job :P
2012-11-01 30604, 2012
2:21 AM
ianmcorvidae
the point here is to ingest things and then have it be a data source for editors
2012-11-01 30629, 2012
2:21 AM
ianmcorvidae
with the ultimate goal being finding mappings so we can create importers
2012-11-01 30646, 2012
2:21 AM
ianmcorvidae
(but also getting some mappings between datasets (including to/from our data) in the process)
2012-11-01 30642, 2012
2:22 AM
reosarevok
ianmcorvidae: sure, but if we import, say, the IA data and data from one label, it would be great if it could tell "heeey, these look the same"!
2012-11-01 30647, 2012
2:22 AM
reosarevok
*!"
2012-11-01 30641, 2012
2:25 AM
ianmcorvidae
that might happen eventually, yeah
2012-11-01 30602, 2012
2:29 AM
ianmcorvidae
ultimately the hope would be that we'd have mappings for both datasets and we'd be able to be like "hm so these look the same in this normalized form"
2012-11-01 30605, 2012
2:31 AM
kurtjx joined the channel
2012-11-01 30625, 2012
3:07 AM
reosarevok
2012-11-01 30631, 2012
3:07 AM
reosarevok
It's... fairly annoying
2012-11-01 30657, 2012
3:44 AM
Prophet5 joined the channel
2012-11-01 30644, 2012
3:54 AM
Prophet5 joined the channel
2012-11-01 30621, 2012
4:31 AM
kepstin-laptop joined the channel
2012-11-01 30643, 2012
5:46 AM
Freso joined the channel
2012-11-01 30655, 2012
6:29 AM
kepstin-laptop joined the channel
2012-11-01 30607, 2012
6:48 AM
Prophet5 joined the channel
2012-11-01 30619, 2012
7:38 AM
Leftmost joined the channel
2012-11-01 30644, 2012
8:34 AM
Prophet5 joined the channel
2012-11-01 30635, 2012
9:04 AM
adhawkins
ocharles: Ping
2012-11-01 30643, 2012
9:04 AM
adhawkins
2012-11-01 30659, 2012
9:10 AM
adhawkins
I didn't download the edit file (just editors). Could it be that? I've never bothered with that one previously.
2012-11-01 30641, 2012
9:14 AM
ianmcorvidae
yeah, there's a bug for that, there's some stuff in the CAA foreign keys file that makes the CAA dump potentially depend on the edit dump
2012-11-01 30603, 2012
9:15 AM
ianmcorvidae
workaround is to comment out the relevant foreign key in the file, or to get the edit dump (I'd recommend the former, personally)
2012-11-01 30618, 2012
9:21 AM
adhawkins
Which file do I edit/
2012-11-01 30619, 2012
9:21 AM
adhawkins
?
2012-11-01 30621, 2012
9:22 AM
ianmcorvidae
admin/sql/caa/CreateFKConstraints.sql lines 12-15
2012-11-01 30631, 2012
9:22 AM
adhawkins
Just comment them out?
2012-11-01 30637, 2012
9:22 AM
ianmcorvidae
yeah
2012-11-01 30651, 2012
9:22 AM
adhawkins
Ok, while you're here, you're responsible for the code that generates the dumps aren't you?
2012-11-01 30655, 2012
9:22 AM
ianmcorvidae
reminding since I don't remember how often you use SQL that '--' is comment in SQL, not #
2012-11-01 30610, 2012
9:23 AM
adhawkins
'Never', so thanks for the reminder :)
2012-11-01 30624, 2012
9:23 AM
ianmcorvidae
inasmuch as I'm in some sense responsible for all the code, at least :)
2012-11-01 30640, 2012
9:23 AM
ianmcorvidae
well, there's a comment up at the top of the file which might have also reminded you, but :)
2012-11-01 30601, 2012
9:24 AM
adhawkins
I'm considering knocking up a script that will download the latest dump (optionally including the edits file).
2012-11-01 30631, 2012
9:24 AM
adhawkins
It'd be a bit easier if instead of a 'latest is xxx' file, there was a file called 'latest', whose contents contained the path.
2012-11-01 30644, 2012
9:24 AM
adhawkins
Then you just wget 'latest', look in the file and wget the rest.
2012-11-01 30658, 2012
9:24 AM
adhawkins
Or can wget do 'ftp://blah/latest-*'?
2012-11-01 30600, 2012
9:25 AM
ianmcorvidae
yeah; I'm not sure why we do that the way we do
2012-11-01 30621, 2012
9:25 AM
ianmcorvidae
wget should be able to do wildcards with ftp urls, yes
2012-11-01 30639, 2012
9:25 AM
ianmcorvidae
of course at present the latest-is file really doesn't do anything except provide a filename you can parse, so :P
2012-11-01 30652, 2012
9:25 AM
adhawkins
Yeah, but I can do that at the shell if necessary.
2012-11-01 30655, 2012
9:25 AM
adhawkins
I'll have a play.
2012-11-01 30636, 2012
9:26 AM
ianmcorvidae
perhaps make a ticket for the changing-the-format thing; I'd like to ask ruaok about it at least, but otherwise I don't see reason not to do that
2012-11-01 30658, 2012
9:26 AM
adhawkins
You could always have both so it's easier for users looking for the latest *and* scripts.
2012-11-01 30616, 2012
9:27 AM
ianmcorvidae
yeah, we'd keep the current latest-is files for compatibility
2012-11-01 30636, 2012
9:27 AM
adhawkins
Yep
2012-11-01 30645, 2012
9:33 AM
Prophet5 joined the channel
2012-11-01 30603, 2012
9:35 AM
adhawkins
Ok, re-running the import. Shame it gets almost to the end before failing :)
2012-11-01 30611, 2012
9:35 AM
adhawkins
What category should that dump ticket be in?
2012-11-01 30637, 2012
9:35 AM
ianmcorvidae
uh
2012-11-01 30639, 2012
9:35 AM
ianmcorvidae
unsure
2012-11-01 30650, 2012
9:35 AM
adhawkins
I'll put it in server for now, someone can move it if necessary.
2012-11-01 30654, 2012
9:35 AM
ianmcorvidae doesn't have our list of components memorized :)
2012-11-01 30602, 2012
9:36 AM
ianmcorvidae
oh, you mean which project? yeah, MBS
2012-11-01 30612, 2012
9:36 AM
adhawkins
Sorry :)
2012-11-01 30640, 2012
9:36 AM
adhawkins
Misc Features?
2012-11-01 30649, 2012
9:36 AM
ianmcorvidae
nah, my fault, should go to bed soon
2012-11-01 30653, 2012
9:36 AM
adhawkins
Scripts?
2012-11-01 30611, 2012
9:37 AM
ianmcorvidae
eh, don't bother with a component for now, none of them look particularly correct
2012-11-01 30611, 2012
9:38 AM
adhawkins
MBS-5541
2012-11-01 30612, 2012
9:38 AM
mb-chat-logger
2012-11-01 30624, 2012
9:38 AM
ianmcorvidae
great, thanks
2012-11-01 30634, 2012
9:51 AM
djce joined the channel
2012-11-01 30620, 2012
9:59 AM
Freso joined the channel
2012-11-01 30649, 2012
10:09 AM
djce joined the channel
2012-11-01 30638, 2012
10:43 AM
adhawkins
ocharles: Ok, data imported, but nothing listening in the VM on port 5000. How do I start up the server?
2012-11-01 30604, 2012
10:44 AM
icrazyhack joined the channel
2012-11-01 30603, 2012
10:56 AM
adhawkins
Ah, think the previous provision might actually have failed.
2012-11-01 30609, 2012
10:56 AM
adhawkins
Will start again (again!) :)
2012-11-01 30619, 2012
11:13 AM
warp
woah. I'm late.
2012-11-01 30604, 2012
11:15 AM
nikki
adhawkins, ianmcorvidae: djce might know, since I'm pretty sure he's the one who created it originally
2012-11-01 30632, 2012
11:15 AM
adhawkins
nikki: This is ocharles new fab and groovy auto-creating VM based on Vagrant and Chef...
2012-11-01 30651, 2012
11:15 AM
nikki
adhawkins: I mean the "latest" file
2012-11-01 30655, 2012
11:15 AM
adhawkins
Ah I see :)
2012-11-01 30607, 2012
11:16 AM
adhawkins
Crossed conversations. Anyway, there's a ticket in place for discussion now.
2012-11-01 30616, 2012
11:16 AM
nikki
I do remember that it's not a symlink, in case the symlink changes halfway through someone downloading
2012-11-01 30641, 2012
11:16 AM
nikki
but I don't remember why it's latest-$timestamp and not latest containing the timestamp...
2012-11-01 30640, 2012
11:17 AM
nikki
probably just that it was done the former way before anyone realised it wasn't the most optimal way
2012-11-01 30612, 2012
11:19 AM
warp
it is the optimal way in the sense that you don't need to perform an extra request
2012-11-01 30653, 2012
11:19 AM
nikki
oh?
2012-11-01 30604, 2012
11:20 AM
warp
(ok, if you hit latest instead of the index it would be the same amount of requests :)
2012-11-01 30608, 2012
11:21 AM
Prophet5 joined the channel
2012-11-01 30636, 2012
11:22 AM
nikki
hm, it looks like my code for getting the timestamp of the latest dump is 8 lines of code when I could do it in 1 if the latest file contained the timestamp (well, 9 and 2 respectively if you include including the relevant modules)
2012-11-01 30657, 2012
11:22 AM
warp
wget -x -m -np `lynx -dump 'http://ftp.musicbrainz.org/pub/musicbrainz/data/fullexport/' | grep 'http://' | awk '{ print $2 }' | grep 'latest-is' | sed 's/latest-is-//'`
2012-11-01 30602, 2012
11:23 AM
warp
that works, though is ugly.
2012-11-01 30619, 2012
11:24 AM
warp
we should just build downloading a full-export into our database provisioning tools :)
2012-11-01 30635, 2012
11:24 AM
nikki
that doesn't help if that's not what you're trying to do
2012-11-01 30638, 2012
11:26 AM
nikki
e.g. the code I just looked at does not come from a script for downloading a full dump
2012-11-01 30640, 2012
11:28 AM
nikki
2012-11-01 30605, 2012
11:30 AM
adhawkins
Great minds eh? :)
2012-11-01 30650, 2012
11:30 AM
nikki
well, I just copied it from trac :P
2012-11-01 30652, 2012
11:30 AM
nikki
but I do agree
2012-11-01 30600, 2012
11:36 AM
adhawkins
./admin/InitDb.pl --createdb --import
2012-11-01 30607, 2012
11:36 AM
adhawkins
Whoops
2012-11-01 30643, 2012
11:36 AM
ocharles
morning
2012-11-01 30656, 2012
11:36 AM
adhawkins
Morning (just!)
2012-11-01 30627, 2012
11:37 AM
ocharles
ya, just...
2012-11-01 30639, 2012
11:37 AM
adhawkins
Oh god, my import is failing again...
2012-11-01 30654, 2012
11:38 AM
adhawkins
2012-11-01 30606, 2012
11:41 AM
adhawkins
Any ideas?
2012-11-01 30649, 2012
11:42 AM
nikki
it looks like there's something wrong with the settings for the postgres user
2012-11-01 30607, 2012
11:43 AM
nikki
but I've not seen that before, so I'm not entirely sure
2012-11-01 30612, 2012
11:43 AM
adhawkins
Grr...it's been working before.
2012-11-01 30616, 2012
11:43 AM
adhawkins
One thing after another! :)
2012-11-01 30656, 2012
11:43 AM
nikki
that's odd then :/
2012-11-01 30617, 2012
11:50 AM
adhawkins
Story of my life :)
2012-11-01 30625, 2012
11:50 AM
adhawkins
Ok, blow the whole thing away and start again (again!)
2012-11-01 30633, 2012
11:50 AM
kurtjx joined the channel
2012-11-01 30655, 2012
12:04 PM
adhawkins
ocharles: When you've got a mo, I'd like to talk to you about a few ideas I've got for the VM, and discuss workflow.
2012-11-01 30626, 2012
12:05 PM
ocharles
sure thing
2012-11-01 30627, 2012
12:05 PM
ocharles
lets talk :)
2012-11-01 30645, 2012
12:05 PM
ocharles
adhawkins: you don't need to start again I wouldn't have thought
2012-11-01 30652, 2012
12:05 PM
ocharles
really 'vagrant provision' should get you to the same place
2012-11-01 30612, 2012
12:06 PM
ocharles
it seems that you just need to restart postgresql because something in provision isn't quite doing that correctly
2012-11-01 30616, 2012
12:06 PM
adhawkins
I think if it's already git cloned, it seems to throw an error.
2012-11-01 30626, 2012
12:06 PM
adhawkins
Ah, restarting postgres, I'll remember that next time :)
2012-11-01 30640, 2012
12:06 PM
adhawkins
What you want to cover first, workflow? Or ideas?
2012-11-01 30602, 2012
12:07 PM
ocharles
whichever is easiest for you
2012-11-01 30614, 2012
12:07 PM
adhawkins
Workflow then.
2012-11-01 30624, 2012
12:07 PM
adhawkins
I've cloned your top level repo, and your cookbooks one.
