#musicbrainz-devel

/

      • kurtjx joined the channel
      • 2012-11-01 30600, 2012

      • kepstin joined the channel
      • 2012-11-01 30651, 2012

      • ocharles
        adhawkins-away: sounds like you're making progress!
      • 2012-11-01 30651, 2012

      • reoafk joined the channel
      • 2012-11-01 30643, 2012

      • kurtjx joined the channel
      • 2012-11-01 30600, 2012

      • reoafk joined the channel
      • 2012-11-01 30638, 2012

      • Freso joined the channel
      • 2012-11-01 30639, 2012

      • ianmcorvidae attempts to install elasticsearch
      • 2012-11-01 30601, 2012

      • Prophet5 joined the channel
      • 2012-11-01 30618, 2012

      • ianmcorvidae
        cool, got ingestr running
      • 2012-11-01 30626, 2012

      • ianmcorvidae
        I mean, it doesn't actually *do* anything since I have no data, but :P
      • 2012-11-01 30640, 2012

      • nikki
        get some data then!
      • 2012-11-01 30634, 2012

      • reosarevok
        You have the IA data!
      • 2012-11-01 30637, 2012

      • reosarevok
        Use it :p
      • 2012-11-01 30631, 2012

      • ianmcorvidae
        at present it really only supports some datasets I don't have :P but yes, that'll be my next step
      • 2012-11-01 30610, 2012

      • reosarevok
        ianmcorvidae: while you're at it, it could probably use some option to add releases in low quality by default :p
      • 2012-11-01 30612, 2012

      • ianmcorvidae
        initially, it won't support adding releases at all :P it needs to get data into its own DB before it can worry about getting it into MB's
      • 2012-11-01 30606, 2012

      • reosarevok
        Ooh
      • 2012-11-01 30607, 2012

      • reosarevok
        cool
      • 2012-11-01 30611, 2012

      • reosarevok
        Import the Naxos Music Library :p
      • 2012-11-01 30602, 2012

      • reosarevok
        Or the BIS site, or Chandos or Hyperion or or or
      • 2012-11-01 30611, 2012

      • reosarevok
        So many things I'd love to see ingested...
      • 2012-11-01 30628, 2012

      • reosarevok goes ingest some chips for now
      • 2012-11-01 30600, 2012

      • ianmcorvidae
        I think I'll start with the IA data :P
      • 2012-11-01 30636, 2012

      • reosarevok
        Sure, sure
      • 2012-11-01 30641, 2012

      • reosarevok
        Ingest it all!
      • 2012-11-01 30656, 2012

      • reosarevok
        does it have any options to find dupes?
      • 2012-11-01 30600, 2012

      • reosarevok
        (from multiple datasets)
      • 2012-11-01 30634, 2012

      • reosarevok
        I guess it doesn't need them yet, but it will at some point
      • 2012-11-01 30645, 2012

      • ianmcorvidae
        that'll be your job :P
      • 2012-11-01 30604, 2012

      • ianmcorvidae
        the point here is to ingest things and then have it be a data source for editors
      • 2012-11-01 30629, 2012

      • ianmcorvidae
        with the ultimate goal being finding mappings so we can create importers
      • 2012-11-01 30646, 2012

      • ianmcorvidae
        (but also getting some mappings between datasets (including to/from our data) in the process)
      • 2012-11-01 30642, 2012

      • reosarevok
        ianmcorvidae: sure, but if we import, say, the IA data and data from one label, it would be great if it could tell "heeey, these look the same"!
      • 2012-11-01 30647, 2012

      • reosarevok
        *!"
      • 2012-11-01 30641, 2012

      • ianmcorvidae
        that might happen eventually, yeah
      • 2012-11-01 30602, 2012

      • ianmcorvidae
        ultimately the hope would be that we'd have mappings for both datasets and we'd be able to be like "hm so these look the same in this normalized form"
      • 2012-11-01 30605, 2012

      • kurtjx joined the channel
      • 2012-11-01 30625, 2012

      • reosarevok
        warp: http://tickets.musicbrainz.org/browse/MBS-5540 could you look into this when you have the time?
      • 2012-11-01 30631, 2012

      • reosarevok
        It's... fairly annoying
      • 2012-11-01 30657, 2012

      • Prophet5 joined the channel
      • 2012-11-01 30644, 2012

      • Prophet5 joined the channel
      • 2012-11-01 30621, 2012

      • kepstin-laptop joined the channel
      • 2012-11-01 30643, 2012

      • Freso joined the channel
      • 2012-11-01 30655, 2012

      • kepstin-laptop joined the channel
      • 2012-11-01 30607, 2012

      • Prophet5 joined the channel
      • 2012-11-01 30619, 2012

      • Leftmost joined the channel
      • 2012-11-01 30644, 2012

      • Prophet5 joined the channel
      • 2012-11-01 30635, 2012

      • adhawkins
        ocharles: Ping
      • 2012-11-01 30643, 2012

      • adhawkins
      • 2012-11-01 30659, 2012

      • adhawkins
        I didn't download the edit file (just editors). Could it be that? I've never bothered with that one previously.
      • 2012-11-01 30641, 2012

      • ianmcorvidae
        yeah, there's a bug for that, there's some stuff in the CAA foreign keys file that makes the CAA dump potentially depend on the edit dump
      • 2012-11-01 30603, 2012

      • ianmcorvidae
        workaround is to comment out the relevant foreign key in the file, or to get the edit dump (I'd recommend the former, personally)
      • 2012-11-01 30618, 2012

      • adhawkins
        Which file do I edit/
      • 2012-11-01 30619, 2012

      • adhawkins
        ?
      • 2012-11-01 30621, 2012

      • ianmcorvidae
        admin/sql/caa/CreateFKConstraints.sql lines 12-15
      • 2012-11-01 30631, 2012

      • adhawkins
        Just comment them out?
      • 2012-11-01 30637, 2012

      • ianmcorvidae
        yeah
      • 2012-11-01 30651, 2012

      • adhawkins
        Ok, while you're here, you're responsible for the code that generates the dumps aren't you?
      • 2012-11-01 30655, 2012

      • ianmcorvidae
        reminding since I don't remember how often you use SQL that '--' is comment in SQL, not #
      • 2012-11-01 30610, 2012

      • adhawkins
        'Never', so thanks for the reminder :)
      • 2012-11-01 30624, 2012

      • ianmcorvidae
        inasmuch as I'm in some sense responsible for all the code, at least :)
      • 2012-11-01 30640, 2012

      • ianmcorvidae
        well, there's a comment up at the top of the file which might have also reminded you, but :)
      • 2012-11-01 30601, 2012

      • adhawkins
        I'm considering knocking up a script that will download the latest dump (optionally including the edits file).
      • 2012-11-01 30631, 2012

      • adhawkins
        It'd be a bit easier if instead of a 'latest is xxx' file, there was a file called 'latest', whose contents contained the path.
      • 2012-11-01 30644, 2012

      • adhawkins
        Then you just wget 'latest', look in the file and wget the rest.
      • 2012-11-01 30658, 2012

      • adhawkins
        Or can wget do 'ftp://blah/latest-*'?
      • 2012-11-01 30600, 2012

      • ianmcorvidae
        yeah; I'm not sure why we do that the way we do
      • 2012-11-01 30621, 2012

      • ianmcorvidae
        wget should be able to do wildcards with ftp urls, yes
      • 2012-11-01 30639, 2012

      • ianmcorvidae
        of course at present the latest-is file really doesn't do anything except provide a filename you can parse, so :P
      • 2012-11-01 30652, 2012

      • adhawkins
        Yeah, but I can do that at the shell if necessary.
      • 2012-11-01 30655, 2012

      • adhawkins
        I'll have a play.
      • 2012-11-01 30636, 2012

      • ianmcorvidae
        perhaps make a ticket for the changing-the-format thing; I'd like to ask ruaok about it at least, but otherwise I don't see reason not to do that
      • 2012-11-01 30658, 2012

      • adhawkins
        You could always have both so it's easier for users looking for the latest *and* scripts.
      • 2012-11-01 30616, 2012

      • ianmcorvidae
        yeah, we'd keep the current latest-is files for compatibility
      • 2012-11-01 30636, 2012

      • adhawkins
        Yep
      • 2012-11-01 30645, 2012

      • Prophet5 joined the channel
      • 2012-11-01 30603, 2012

      • adhawkins
        Ok, re-running the import. Shame it gets almost to the end before failing :)
      • 2012-11-01 30611, 2012

      • adhawkins
        What category should that dump ticket be in?
      • 2012-11-01 30637, 2012

      • ianmcorvidae
        uh
      • 2012-11-01 30639, 2012

      • ianmcorvidae
        unsure
      • 2012-11-01 30650, 2012

      • adhawkins
        I'll put it in server for now, someone can move it if necessary.
      • 2012-11-01 30654, 2012

      • ianmcorvidae doesn't have our list of components memorized :)
      • 2012-11-01 30602, 2012

      • ianmcorvidae
        oh, you mean which project? yeah, MBS
      • 2012-11-01 30612, 2012

      • adhawkins
        Sorry :)
      • 2012-11-01 30640, 2012

      • adhawkins
        Misc Features?
      • 2012-11-01 30649, 2012

      • ianmcorvidae
        nah, my fault, should go to bed soon
      • 2012-11-01 30653, 2012

      • adhawkins
        Scripts?
      • 2012-11-01 30611, 2012

      • ianmcorvidae
        eh, don't bother with a component for now, none of them look particularly correct
      • 2012-11-01 30611, 2012

      • adhawkins
        MBS-5541
      • 2012-11-01 30612, 2012

      • mb-chat-logger
      • 2012-11-01 30624, 2012

      • ianmcorvidae
        great, thanks
      • 2012-11-01 30634, 2012

      • djce joined the channel
      • 2012-11-01 30620, 2012

      • Freso joined the channel
      • 2012-11-01 30649, 2012

      • djce joined the channel
      • 2012-11-01 30638, 2012

      • adhawkins
        ocharles: Ok, data imported, but nothing listening in the VM on port 5000. How do I start up the server?
      • 2012-11-01 30604, 2012

      • icrazyhack joined the channel
      • 2012-11-01 30603, 2012

      • adhawkins
        Ah, think the previous provision might actually have failed.
      • 2012-11-01 30609, 2012

      • adhawkins
        Will start again (again!) :)
      • 2012-11-01 30619, 2012

      • warp
        woah. I'm late.
      • 2012-11-01 30604, 2012

      • nikki
        adhawkins, ianmcorvidae: djce might know, since I'm pretty sure he's the one who created it originally
      • 2012-11-01 30632, 2012

      • adhawkins
        nikki: This is ocharles new fab and groovy auto-creating VM based on Vagrant and Chef...
      • 2012-11-01 30651, 2012

      • nikki
        adhawkins: I mean the "latest" file
      • 2012-11-01 30655, 2012

      • adhawkins
        Ah I see :)
      • 2012-11-01 30607, 2012

      • adhawkins
        Crossed conversations. Anyway, there's a ticket in place for discussion now.
      • 2012-11-01 30616, 2012

      • nikki
        I do remember that it's not a symlink, in case the symlink changes halfway through someone downloading
      • 2012-11-01 30641, 2012

      • nikki
        but I don't remember why it's latest-$timestamp and not latest containing the timestamp...
      • 2012-11-01 30640, 2012

      • nikki
        probably just that it was done the former way before anyone realised it wasn't the most optimal way
      • 2012-11-01 30612, 2012

      • warp
        it is the optimal way in the sense that you don't need to perform an extra request
      • 2012-11-01 30653, 2012

      • nikki
        oh?
      • 2012-11-01 30604, 2012

      • warp
        (ok, if you hit latest instead of the index it would be the same amount of requests :)
      • 2012-11-01 30608, 2012

      • Prophet5 joined the channel
      • 2012-11-01 30636, 2012

      • nikki
        hm, it looks like my code for getting the timestamp of the latest dump is 8 lines of code when I could do it in 1 if the latest file contained the timestamp (well, 9 and 2 respectively if you include including the relevant modules)
      • 2012-11-01 30657, 2012

      • warp
        wget -x -m -np `lynx -dump 'http://ftp.musicbrainz.org/pub/musicbrainz/data/fullexport/' | grep 'http://' | awk '{ print $2 }' | grep 'latest-is' | sed 's/latest-is-//'`
      • 2012-11-01 30602, 2012

      • warp
        that works, though is ugly.
      • 2012-11-01 30619, 2012

      • warp
        we should just build downloading a full-export into our database provisioning tools :)
      • 2012-11-01 30635, 2012

      • nikki
        that doesn't help if that's not what you're trying to do
      • 2012-11-01 30638, 2012

      • nikki
        e.g. the code I just looked at does not come from a script for downloading a full dump
      • 2012-11-01 30640, 2012

      • nikki
      • 2012-11-01 30605, 2012

      • adhawkins
        Great minds eh? :)
      • 2012-11-01 30650, 2012

      • nikki
        well, I just copied it from trac :P
      • 2012-11-01 30652, 2012

      • nikki
        but I do agree
      • 2012-11-01 30600, 2012

      • adhawkins
        ./admin/InitDb.pl --createdb --import
      • 2012-11-01 30607, 2012

      • adhawkins
        Whoops
      • 2012-11-01 30643, 2012

      • ocharles
        morning
      • 2012-11-01 30656, 2012

      • adhawkins
        Morning (just!)
      • 2012-11-01 30627, 2012

      • ocharles
        ya, just...
      • 2012-11-01 30639, 2012

      • adhawkins
        Oh god, my import is failing again...
      • 2012-11-01 30654, 2012

      • adhawkins
      • 2012-11-01 30606, 2012

      • adhawkins
        Any ideas?
      • 2012-11-01 30649, 2012

      • nikki
        it looks like there's something wrong with the settings for the postgres user
      • 2012-11-01 30607, 2012

      • nikki
        but I've not seen that before, so I'm not entirely sure
      • 2012-11-01 30612, 2012

      • adhawkins
        Grr...it's been working before.
      • 2012-11-01 30616, 2012

      • adhawkins
        One thing after another! :)
      • 2012-11-01 30656, 2012

      • nikki
        that's odd then :/
      • 2012-11-01 30617, 2012

      • adhawkins
        Story of my life :)
      • 2012-11-01 30625, 2012

      • adhawkins
        Ok, blow the whole thing away and start again (again!)
      • 2012-11-01 30633, 2012

      • kurtjx joined the channel
      • 2012-11-01 30655, 2012

      • adhawkins
        ocharles: When you've got a mo, I'd like to talk to you about a few ideas I've got for the VM, and discuss workflow.
      • 2012-11-01 30626, 2012

      • ocharles
        sure thing
      • 2012-11-01 30627, 2012

      • ocharles
        lets talk :)
      • 2012-11-01 30645, 2012

      • ocharles
        adhawkins: you don't need to start again I wouldn't have thought
      • 2012-11-01 30652, 2012

      • ocharles
        really 'vagrant provision' should get you to the same place
      • 2012-11-01 30612, 2012

      • ocharles
        it seems that you just need to restart postgresql because something in provision isn't quite doing that correctly
      • 2012-11-01 30616, 2012

      • adhawkins
        I think if it's already git cloned, it seems to throw an error.
      • 2012-11-01 30626, 2012

      • adhawkins
        Ah, restarting postgres, I'll remember that next time :)
      • 2012-11-01 30640, 2012

      • adhawkins
        What you want to cover first, workflow? Or ideas?
      • 2012-11-01 30602, 2012

      • ocharles
        whichever is easiest for you
      • 2012-11-01 30614, 2012

      • adhawkins
        Workflow then.
      • 2012-11-01 30624, 2012

      • adhawkins
        I've cloned your top level repo, and your cookbooks one.