#musicbrainz-devel

/

      • Freso joined the channel
      • 2012-11-01 30655, 2012

      • kepstin-laptop joined the channel
      • 2012-11-01 30607, 2012

      • Prophet5 joined the channel
      • 2012-11-01 30619, 2012

      • Leftmost joined the channel
      • 2012-11-01 30644, 2012

      • Prophet5 joined the channel
      • 2012-11-01 30635, 2012

      • adhawkins
        ocharles: Ping
      • 2012-11-01 30643, 2012

      • adhawkins
      • 2012-11-01 30659, 2012

      • adhawkins
        I didn't download the edit file (just editors). Could it be that? I've never bothered with that one previously.
      • 2012-11-01 30641, 2012

      • ianmcorvidae
        yeah, there's a bug for that, there's some stuff in the CAA foreign keys file that makes the CAA dump potentially depend on the edit dump
      • 2012-11-01 30603, 2012

      • ianmcorvidae
        workaround is to comment out the relevant foreign key in the file, or to get the edit dump (I'd recommend the former, personally)
      • 2012-11-01 30618, 2012

      • adhawkins
        Which file do I edit/
      • 2012-11-01 30619, 2012

      • adhawkins
        ?
      • 2012-11-01 30621, 2012

      • ianmcorvidae
        admin/sql/caa/CreateFKConstraints.sql lines 12-15
      • 2012-11-01 30631, 2012

      • adhawkins
        Just comment them out?
      • 2012-11-01 30637, 2012

      • ianmcorvidae
        yeah
      • 2012-11-01 30651, 2012

      • adhawkins
        Ok, while you're here, you're responsible for the code that generates the dumps aren't you?
      • 2012-11-01 30655, 2012

      • ianmcorvidae
        reminding since I don't remember how often you use SQL that '--' is comment in SQL, not #
      • 2012-11-01 30610, 2012

      • adhawkins
        'Never', so thanks for the reminder :)
      • 2012-11-01 30624, 2012

      • ianmcorvidae
        inasmuch as I'm in some sense responsible for all the code, at least :)
      • 2012-11-01 30640, 2012

      • ianmcorvidae
        well, there's a comment up at the top of the file which might have also reminded you, but :)
      • 2012-11-01 30601, 2012

      • adhawkins
        I'm considering knocking up a script that will download the latest dump (optionally including the edits file).
      • 2012-11-01 30631, 2012

      • adhawkins
        It'd be a bit easier if instead of a 'latest is xxx' file, there was a file called 'latest', whose contents contained the path.
      • 2012-11-01 30644, 2012

      • adhawkins
        Then you just wget 'latest', look in the file and wget the rest.
      • 2012-11-01 30658, 2012

      • adhawkins
        Or can wget do 'ftp://blah/latest-*'?
      • 2012-11-01 30600, 2012

      • ianmcorvidae
        yeah; I'm not sure why we do that the way we do
      • 2012-11-01 30621, 2012

      • ianmcorvidae
        wget should be able to do wildcards with ftp urls, yes
      • 2012-11-01 30639, 2012

      • ianmcorvidae
        of course at present the latest-is file really doesn't do anything except provide a filename you can parse, so :P
      • 2012-11-01 30652, 2012

      • adhawkins
        Yeah, but I can do that at the shell if necessary.
      • 2012-11-01 30655, 2012

      • adhawkins
        I'll have a play.
      • 2012-11-01 30636, 2012

      • ianmcorvidae
        perhaps make a ticket for the changing-the-format thing; I'd like to ask ruaok about it at least, but otherwise I don't see reason not to do that
      • 2012-11-01 30658, 2012

      • adhawkins
        You could always have both so it's easier for users looking for the latest *and* scripts.
      • 2012-11-01 30616, 2012

      • ianmcorvidae
        yeah, we'd keep the current latest-is files for compatibility
      • 2012-11-01 30636, 2012

      • adhawkins
        Yep
      • 2012-11-01 30645, 2012

      • Prophet5 joined the channel
      • 2012-11-01 30603, 2012

      • adhawkins
        Ok, re-running the import. Shame it gets almost to the end before failing :)
      • 2012-11-01 30611, 2012

      • adhawkins
        What category should that dump ticket be in?
      • 2012-11-01 30637, 2012

      • ianmcorvidae
        uh
      • 2012-11-01 30639, 2012

      • ianmcorvidae
        unsure
      • 2012-11-01 30650, 2012

      • adhawkins
        I'll put it in server for now, someone can move it if necessary.
      • 2012-11-01 30654, 2012

      • ianmcorvidae doesn't have our list of components memorized :)
      • 2012-11-01 30602, 2012

      • ianmcorvidae
        oh, you mean which project? yeah, MBS
      • 2012-11-01 30612, 2012

      • adhawkins
        Sorry :)
      • 2012-11-01 30640, 2012

      • adhawkins
        Misc Features?
      • 2012-11-01 30649, 2012

      • ianmcorvidae
        nah, my fault, should go to bed soon
      • 2012-11-01 30653, 2012

      • adhawkins
        Scripts?
      • 2012-11-01 30611, 2012

      • ianmcorvidae
        eh, don't bother with a component for now, none of them look particularly correct
      • 2012-11-01 30611, 2012

      • adhawkins
        MBS-5541
      • 2012-11-01 30612, 2012

      • mb-chat-logger
      • 2012-11-01 30624, 2012

      • ianmcorvidae
        great, thanks
      • 2012-11-01 30634, 2012

      • djce joined the channel
      • 2012-11-01 30620, 2012

      • Freso joined the channel
      • 2012-11-01 30649, 2012

      • djce joined the channel
      • 2012-11-01 30638, 2012

      • adhawkins
        ocharles: Ok, data imported, but nothing listening in the VM on port 5000. How do I start up the server?
      • 2012-11-01 30604, 2012

      • icrazyhack joined the channel
      • 2012-11-01 30603, 2012

      • adhawkins
        Ah, think the previous provision might actually have failed.
      • 2012-11-01 30609, 2012

      • adhawkins
        Will start again (again!) :)
      • 2012-11-01 30619, 2012

      • warp
        woah. I'm late.
      • 2012-11-01 30604, 2012

      • nikki
        adhawkins, ianmcorvidae: djce might know, since I'm pretty sure he's the one who created it originally
      • 2012-11-01 30632, 2012

      • adhawkins
        nikki: This is ocharles new fab and groovy auto-creating VM based on Vagrant and Chef...
      • 2012-11-01 30651, 2012

      • nikki
        adhawkins: I mean the "latest" file
      • 2012-11-01 30655, 2012

      • adhawkins
        Ah I see :)
      • 2012-11-01 30607, 2012

      • adhawkins
        Crossed conversations. Anyway, there's a ticket in place for discussion now.
      • 2012-11-01 30616, 2012

      • nikki
        I do remember that it's not a symlink, in case the symlink changes halfway through someone downloading
      • 2012-11-01 30641, 2012

      • nikki
        but I don't remember why it's latest-$timestamp and not latest containing the timestamp...
      • 2012-11-01 30640, 2012

      • nikki
        probably just that it was done the former way before anyone realised it wasn't the most optimal way
      • 2012-11-01 30612, 2012

      • warp
        it is the optimal way in the sense that you don't need to perform an extra request
      • 2012-11-01 30653, 2012

      • nikki
        oh?
      • 2012-11-01 30604, 2012

      • warp
        (ok, if you hit latest instead of the index it would be the same amount of requests :)
      • 2012-11-01 30608, 2012

      • Prophet5 joined the channel
      • 2012-11-01 30636, 2012

      • nikki
        hm, it looks like my code for getting the timestamp of the latest dump is 8 lines of code when I could do it in 1 if the latest file contained the timestamp (well, 9 and 2 respectively if you include including the relevant modules)
      • 2012-11-01 30657, 2012

      • warp
        wget -x -m -np `lynx -dump 'http://ftp.musicbrainz.org/pub/musicbrainz/data/fullexport/' | grep 'http://' | awk '{ print $2 }' | grep 'latest-is' | sed 's/latest-is-//'`
      • 2012-11-01 30602, 2012

      • warp
        that works, though is ugly.
      • 2012-11-01 30619, 2012

      • warp
        we should just build downloading a full-export into our database provisioning tools :)
      • 2012-11-01 30635, 2012

      • nikki
        that doesn't help if that's not what you're trying to do
      • 2012-11-01 30638, 2012

      • nikki
        e.g. the code I just looked at does not come from a script for downloading a full dump
      • 2012-11-01 30640, 2012

      • nikki
      • 2012-11-01 30605, 2012

      • adhawkins
        Great minds eh? :)
      • 2012-11-01 30650, 2012

      • nikki
        well, I just copied it from trac :P
      • 2012-11-01 30652, 2012

      • nikki
        but I do agree
      • 2012-11-01 30600, 2012

      • adhawkins
        ./admin/InitDb.pl --createdb --import
      • 2012-11-01 30607, 2012

      • adhawkins
        Whoops
      • 2012-11-01 30643, 2012

      • ocharles
        morning
      • 2012-11-01 30656, 2012

      • adhawkins
        Morning (just!)
      • 2012-11-01 30627, 2012

      • ocharles
        ya, just...
      • 2012-11-01 30639, 2012

      • adhawkins
        Oh god, my import is failing again...
      • 2012-11-01 30654, 2012

      • adhawkins
      • 2012-11-01 30606, 2012

      • adhawkins
        Any ideas?
      • 2012-11-01 30649, 2012

      • nikki
        it looks like there's something wrong with the settings for the postgres user
      • 2012-11-01 30607, 2012

      • nikki
        but I've not seen that before, so I'm not entirely sure
      • 2012-11-01 30612, 2012

      • adhawkins
        Grr...it's been working before.
      • 2012-11-01 30616, 2012

      • adhawkins
        One thing after another! :)
      • 2012-11-01 30656, 2012

      • nikki
        that's odd then :/
      • 2012-11-01 30617, 2012

      • adhawkins
        Story of my life :)
      • 2012-11-01 30625, 2012

      • adhawkins
        Ok, blow the whole thing away and start again (again!)
      • 2012-11-01 30633, 2012

      • kurtjx joined the channel
      • 2012-11-01 30655, 2012

      • adhawkins
        ocharles: When you've got a mo, I'd like to talk to you about a few ideas I've got for the VM, and discuss workflow.
      • 2012-11-01 30626, 2012

      • ocharles
        sure thing
      • 2012-11-01 30627, 2012

      • ocharles
        lets talk :)
      • 2012-11-01 30645, 2012

      • ocharles
        adhawkins: you don't need to start again I wouldn't have thought
      • 2012-11-01 30652, 2012

      • ocharles
        really 'vagrant provision' should get you to the same place
      • 2012-11-01 30612, 2012

      • ocharles
        it seems that you just need to restart postgresql because something in provision isn't quite doing that correctly
      • 2012-11-01 30616, 2012

      • adhawkins
        I think if it's already git cloned, it seems to throw an error.
      • 2012-11-01 30626, 2012

      • adhawkins
        Ah, restarting postgres, I'll remember that next time :)
      • 2012-11-01 30640, 2012

      • adhawkins
        What you want to cover first, workflow? Or ideas?
      • 2012-11-01 30602, 2012

      • ocharles
        whichever is easiest for you
      • 2012-11-01 30614, 2012

      • adhawkins
        Workflow then.
      • 2012-11-01 30624, 2012

      • adhawkins
        I've cloned your top level repo, and your cookbooks one.
      • 2012-11-01 30630, 2012

      • adhawkins
        I've also created a branch in each.
      • 2012-11-01 30649, 2012

      • adhawkins
        If I do any work in either, then I guess I can just submit pull requests via Github for your review.
      • 2012-11-01 30653, 2012

      • adhawkins
        That sounds like a good start?
      • 2012-11-01 30604, 2012

      • adhawkins
        Other option is for me to create branches directly in your repo, and you merge from there.
      • 2012-11-01 30613, 2012

      • ocharles
        nah, pull requests sound fine
      • 2012-11-01 30626, 2012

      • adhawkins
        Ok. Bit more hassle for me due to the submodules, but hopefully it'll work out!
      • 2012-11-01 30631, 2012

      • adhawkins
        We'll give that a go then.
      • 2012-11-01 30634, 2012

      • ocharles
        oh?
      • 2012-11-01 30641, 2012

      • ocharles
        because you have to bump the submodule up too?
      • 2012-11-01 30649, 2012

      • ocharles
        I can take care of that if you give me instructions in the cookbooks pull request
      • 2012-11-01 30654, 2012

      • reosarevok joined the channel
      • 2012-11-01 30659, 2012

      • adhawkins
        You have to move the submodule in the fork to point to the forked cookbook
      • 2012-11-01 30611, 2012

      • adhawkins
        Does that make sense? :)
      • 2012-11-01 30626, 2012

      • ocharles
        yea, but as I said - I can do that juggling
      • 2012-11-01 30646, 2012

      • adhawkins
        But for me to work I have to do it too. Otherwise my 'master' repo would be pointing to your cookbook
      • 2012-11-01 30635, 2012

      • adhawkins
        Anyway, a couple of ideas to move it forward.
      • 2012-11-01 30609, 2012

      • adhawkins
        The first we've already talked about. Create a 'base' VM with (say) a 10G 'root' partition, and a bigger (what size?) partition for the postgres databases.
      • 2012-11-01 30621, 2012

      • adhawkins
        I can proceed with that if you tell me what OS you want to base it on.
      • 2012-11-01 30645, 2012

      • kurtjx joined the channel
      • 2012-11-01 30656, 2012

      • adhawkins
        Other idea was to write a script that downloads the latest DB dumps, and include this in the VM to make it easier for people to do the initial import
      • 2012-11-01 30607, 2012

      • ocharles
        brb, gotta chat with last.fm
      • 2012-11-01 30635, 2012

      • adhawkins
        ok
      • 2012-11-01 30622, 2012

      • Freso joined the channel
      • 2012-11-01 30621, 2012

      • adhawkins
        ocharles: On this VM, nothing appears to be listening on port 5000. What process should be running for that?
      • 2012-11-01 30618, 2012

      • ocharles
        hmm, plackup
      • 2012-11-01 30632, 2012

      • ocharles
        you should have /etc/service/musicbrainz-server
      • 2012-11-01 30602, 2012

      • adhawkins
        Ok, and what starts that?
      • 2012-11-01 30612, 2012

      • adhawkins
        I'll do a reboot after importing and see if that helps
      • 2012-11-01 30615, 2012

      • reosarevok
        Why are the forums a 504?
      • 2012-11-01 30627, 2012

      • Freso
        Sh*t'n'giggles?
      • 2012-11-01 30601, 2012

      • kurtjx joined the channel
      • 2012-11-01 30619, 2012

      • reosarevok
        Has anyone seen ijabz lately?
      • 2012-11-01 30650, 2012

      • ocharles
        adhawkins: sudo svc -u /etc/service/musicbrainz-server should start it
      • 2012-11-01 30601, 2012

      • ocharles
        you can check /etc/service/musicbrainz-server/log/main/current to see what it's doing
      • 2012-11-01 30611, 2012

      • ocharles
        damn I write some sexy code sometimes https://gist.github.com/654699e05dc612212f7a
      • 2012-11-01 30612, 2012

      • warp
        meh, the coverart_redirect code is ugly.
      • 2012-11-01 30629, 2012

      • luks
        warp: renaming README files in the postgresql extensions broke the Makefiles
      • 2012-11-01 30655, 2012

      • warp
        luks: oh, the READMEs get installed?
      • 2012-11-01 30613, 2012

      • luks
        yes
      • 2012-11-01 30617, 2012

      • ocharles
        I should add those extensions to jenkins
      • 2012-11-01 30620, 2012

      • luks
        that's why they are in makefiles :)
      • 2012-11-01 30601, 2012

      • luks
        and can you please remove the libunac mention in the unaccent one
      • 2012-11-01 30614, 2012

      • luks
        I'm not sure why it's there, but libunac is definitely not required