#musicbrainz-devel

/

      • ianmcorvidae
        but whoo
      • 2014-10-09 28208, 2014

      • alastairp
        yeah, unfortunately very little way to make it faster
      • 2014-10-09 28212, 2014

      • alastairp
        just do lots of cores
      • 2014-10-09 28217, 2014

      • alastairp
        run it on as many machines as you can!
      • 2014-10-09 28233, 2014

      • ianmcorvidae
        heh, I don't have parallel installed at present
      • 2014-10-09 28251, 2014

      • ianmcorvidae
        will it do the right thing if I just run a bunch of copies of it?
      • 2014-10-09 28206, 2014

      • alastairp
        yep
      • 2014-10-09 28209, 2014

      • alastairp
        no
      • 2014-10-09 28211, 2014

      • alastairp
        sorry :(
      • 2014-10-09 28217, 2014

      • ianmcorvidae
        heh, it's having some issues with 24bit or 96k
      • 2014-10-09 28223, 2014

      • ianmcorvidae
        not sure which one it is, but
      • 2014-10-09 28234, 2014

      • alastairp
        because I write to the process log every time it does a file
      • 2014-10-09 28241, 2014

      • alastairp
        um, we should work out something better there
      • 2014-10-09 28259, 2014

      • alastairp
        only write to file when you ^c or it crashes, and then only append
      • 2014-10-09 28216, 2014

      • alastairp
        hmm. essentia is? try and run just the extractor? if so, please file a bug on mtg/essentia
      • 2014-10-09 28221, 2014

      • kepstin-laptop
        yeah, to really get this working multiprocessing properly, you should be doing it in-process, with the log writing in a single spot, imo.
      • 2014-10-09 28229, 2014

      • alastairp
        kepstin-laptop: tomorrow, tomorrow
      • 2014-10-09 28216, 2014

      • ianmcorvidae
        yeah, it seems to think this 24bit/96k file is 44100 and has nothing about simple size :(
      • 2014-10-09 28221, 2014

      • ianmcorvidae
        well, it'll need rerunning I guess
      • 2014-10-09 28227, 2014

      • kepstin-laptop
        huh, the streaming_extractor_music seems to be not making progress on some of my files
      • 2014-10-09 28240, 2014

      • kepstin-laptop
        been hanging for several minutes now
      • 2014-10-09 28246, 2014

      • bitmap
        upgrading taglib did something, since now it's progressed to a SIGSEGV :)
      • 2014-10-09 28231, 2014

      • ianmcorvidae
        alastairp: for reference at some point in the future (or tell me where to file a bug!): https://gist.github.com/ianmcorvidae/5d7a61b8a0a1… which produces http://acousticbrainz.org/9a1aedc4-88ce-407e-a7d8… which claims analysis_sample_rate of 44100 for a 96kHz flac, no mention of the 24bit audio samples though I bet something funny's going on there too :)
      • 2014-10-09 28240, 2014

      • ianmcorvidae
        can provide the file for testing too if needed
      • 2014-10-09 28259, 2014

      • ianmcorvidae loves throwing 24bit/96kHz files at things, they're so good at breaking everything :P
      • 2014-10-09 28205, 2014

      • alastairp
        ianmcorvidae: it looks like it downsampled
      • 2014-10-09 28226, 2014

      • ianmcorvidae
        ah, hm
      • 2014-10-09 28239, 2014

      • ianmcorvidae
        well, then, perhaps nonoptimal but at least not broken
      • 2014-10-09 28241, 2014

      • alastairp
        analysis sample rate is an option to the processor
      • 2014-10-09 28204, 2014

      • alastairp
        which you could probably set if you wanted to
      • 2014-10-09 28206, 2014

      • kepstin-laptop
        does the essentia stuff require a specific sample rate, or does it support various ones?
      • 2014-10-09 28236, 2014

      • alastairp
        I believe the latter
      • 2014-10-09 28214, 2014

      • kepstin-laptop
        replaygain was kind of annoying that way; most implementations used precomputed tables which they only had up to either 44.1 or 48kHz
      • 2014-10-09 28239, 2014

      • ianmcorvidae
        yeah, IIRC my 96kHz files still aren't replaygained
      • 2014-10-09 28243, 2014

      • ianmcorvidae
        because metaflac is such an implementation :P
      • 2014-10-09 28207, 2014

      • alastairp
        yep, there's a resampler there to take it down to 44.1k
      • 2014-10-09 28218, 2014

      • kepstin-laptop
        foobar2000 can replaygain them, I think; it uses an r.128-based algorithm rather than the original replaygain algorithm.
      • 2014-10-09 28204, 2014

      • ianmcorvidae wonders what happens when there are multiple submissions server-side
      • 2014-10-09 28211, 2014

      • alastairp
        it stores all of them
      • 2014-10-09 28217, 2014

      • alastairp
        we still don't know how to sort them out
      • 2014-10-09 28220, 2014

      • alastairp
        but it'll happen later
      • 2014-10-09 28224, 2014

      • ianmcorvidae
        right
      • 2014-10-09 28235, 2014

      • ianmcorvidae
        which one appears at the webservice endpoint, if you know?
      • 2014-10-09 28236, 2014

      • alastairp
        come on guys, only 500 tracks. you're slowing down
      • 2014-10-09 28248, 2014

      • alastairp
        good question. ask the boss
      • 2014-10-09 28217, 2014

      • ianmcorvidae just reads code, lol
      • 2014-10-09 28219, 2014

      • ianmcorvidae
        looks like it doesn't actually keep everything, it only keeps it if there's nothing or if it's a lossless submit of something that there's only lossy for
      • 2014-10-09 28224, 2014

      • ianmcorvidae
        so that answers that
      • 2014-10-09 28230, 2014

      • kepstin-laptop
        actually, it is slowing down
      • 2014-10-09 28237, 2014

      • ianmcorvidae
      • 2014-10-09 28246, 2014

      • kepstin-laptop
        it hasn't finished processing a new track for me in almost 20 minutes
      • 2014-10-09 28212, 2014

      • ianmcorvidae
        there's a comment in the code about something to do with logging and writing to the file, maybe that?
      • 2014-10-09 28225, 2014

      • alastairp
        oh yeah. does it really take that long?!
      • 2014-10-09 28236, 2014

      • alastairp
        kepstin-laptop: have you tried to run the file with just the extractor
      • 2014-10-09 28243, 2014

      • ianmcorvidae
        seems weird that it would, I guess, but
      • 2014-10-09 28245, 2014

      • alastairp
        I hope it's not my super-dodgy file logger
      • 2014-10-09 28251, 2014

      • kepstin-laptop
        i've had 4 streaming_extractor_music processes running for about 20 minutes each now
      • 2014-10-09 28254, 2014

      • ianmcorvidae
        mine is hanging on "compute audio features" now
      • 2014-10-09 28211, 2014

      • ianmcorvidae kills it and restarts to see if it's just hanging on something mysterious
      • 2014-10-09 28238, 2014

      • kepstin-laptop
        and the default build doesn't include debug info, so checking with gdb is giving nothing useful :/
      • 2014-10-09 28200, 2014

      • ruaok_ joined the channel
      • 2014-10-09 28205, 2014

      • ruaok_ joined the channel
      • 2014-10-09 28223, 2014

      • ianmcorvidae is getting the same hanging on mine when I run it directly
      • 2014-10-09 28231, 2014

      • ruaok_lappy
      • 2014-10-09 28238, 2014

      • ruaok_lappy
        installing libswreample didn’t help me.
      • 2014-10-09 28247, 2014

      • ruaok_lappy
        does the waf program cache its results?
      • 2014-10-09 28208, 2014

      • ianmcorvidae
        ruaok_lappy: libsamplerate0-dev according to http://essentia.upf.edu/documentation/installing.…
      • 2014-10-09 28233, 2014

      • ianmcorvidae
        though nikki was saying something about a different thing
      • 2014-10-09 28257, 2014

      • ruaok_lappy
        I installed both now and still no go. :(
      • 2014-10-09 28200, 2014

      • kepstin-laptop
        alastairp: yeah, the streaming_extractor_music tool is just hanging in the "Compute audio features" step on some tracks.
      • 2014-10-09 28208, 2014

      • ianmcorvidae
        is your taglib recent enough?
      • 2014-10-09 28214, 2014

      • kepstin-laptop has a few that reliably reproduce the issue
      • 2014-10-09 28222, 2014

      • ruaok_lappy
        I haven’t gotten to that bridg yet.
      • 2014-10-09 28225, 2014

      • ruaok_lappy
        is that via pip?
      • 2014-10-09 28232, 2014

      • ruaok_lappy has the ve already setup
      • 2014-10-09 28232, 2014

      • ianmcorvidae
        no, it's a C library
      • 2014-10-09 28241, 2014

      • ianmcorvidae
        and it's required by essentia
      • 2014-10-09 28242, 2014

      • ruaok_lappy
        oh right, not in packages.
      • 2014-10-09 28246, 2014

      • ruaok_lappy
        k
      • 2014-10-09 28248, 2014

      • ianmcorvidae
        no, that's the python one
      • 2014-10-09 28200, 2014

      • ianmcorvidae
        the C one is in packages, but if you're running an old ubuntu it's not recent enough
      • 2014-10-09 28203, 2014

      • ianmcorvidae
        which was nikki's issue
      • 2014-10-09 28218, 2014

      • ruaok_lappy
        I’m on 14.04
      • 2014-10-09 28232, 2014

      • ianmcorvidae
        okay, that one should be fine
      • 2014-10-09 28238, 2014

      • ianmcorvidae
        what error are you getting, then?
      • 2014-10-09 28218, 2014

      • ruaok_lappy
        I hadn’t made it that far down the list of waf complaints yet. installing now.
      • 2014-10-09 28249, 2014

      • ianmcorvidae
        and yeah, this file just doesn't work with essentia :(
      • 2014-10-09 28239, 2014

      • ruaok_lappy
        libtagc0-dev on u 14.04 makes waf happy.
      • 2014-10-09 28244, 2014

      • ianmcorvidae
        right, that's one of the taglib things
      • 2014-10-09 28250, 2014

      • ianmcorvidae
        so now we know it's that one, anyway
      • 2014-10-09 28256, 2014

      • ianmcorvidae
        I guess that's the dev package for the C bindings
      • 2014-10-09 28257, 2014

      • ianmcorvidae
        or something
      • 2014-10-09 28243, 2014

      • kepstin-laptop
        well, so much for me submitting my entire music collection; something inside the streaming_extractor_music tools is going into an infinite loopo on several of my files :/
      • 2014-10-09 28223, 2014

      • ruaok
        you suck.
      • 2014-10-09 28227, 2014

      • ruaok
        oh wait, you're not reo.
      • 2014-10-09 28229, 2014

      • ruaok
        :-D
      • 2014-10-09 28230, 2014

      • ianmcorvidae
        kepstin-laptop: you can add things manually to ~/.abzsubmit.log to have it skip them, btw
      • 2014-10-09 28202, 2014

      • kepstin-laptop
        true, but my log is useless due to the parallel stuff anyways, and it'll hang for hours before i notice and add a file to it and restart processing :/
      • 2014-10-09 28209, 2014

      • ianmcorvidae
        yeah :/
      • 2014-10-09 28229, 2014

      • ianmcorvidae isn't running in parallel, but the bits with actually noticing still apply
      • 2014-10-09 28240, 2014

      • reosarevok
        ruaok: well he was closer to playing the vacuum cleaner than I was, for what it's worth!
      • 2014-10-09 28243, 2014

      • reosarevok
        ;)
      • 2014-10-09 28245, 2014

      • alastairp
        kepstin-laptop: cool, can you submit a bug
      • 2014-10-09 28249, 2014

      • alastairp
        and get the files to me
      • 2014-10-09 28253, 2014

      • alastairp
        and I'll pass them on
      • 2014-10-09 28254, 2014

      • ruaok_lappy
        :)
      • 2014-10-09 28218, 2014

      • ianmcorvidae has at least one failing file as well, should it be wanted
      • 2014-10-09 28225, 2014

      • bitmap
        yay, it works after recompiling everything that depended on taglib
      • 2014-10-09 28245, 2014

      • alastairp
        doh
      • 2014-10-09 28251, 2014

      • alastairp
        ianmcorvidae: yep, for sure!
      • 2014-10-09 28227, 2014

      • alastairp
        kepstin-laptop: you can CFLAGS=-g ./waf etc etc
      • 2014-10-09 28200, 2014

      • ianmcorvidae
        alastairp: preferred method for getting it to you?
      • 2014-10-09 28211, 2014

      • ianmcorvidae
        (19M flac)
      • 2014-10-09 28223, 2014

      • alastairp
        put it somehwere I can download it
      • 2014-10-09 28223, 2014

      • alastairp
        ruaok_lappy: weird, got it working?
      • 2014-10-09 28232, 2014

      • ruaok_lappy
        on track so far.
      • 2014-10-09 28237, 2014

      • nikki
        I got essentia to compile in a vm
      • 2014-10-09 28241, 2014

      • nikki
        how do I use the client thing?
      • 2014-10-09 28203, 2014

      • ianmcorvidae
        nikki: get dependencies (virtualenv, I guess) and then ./abzsubmit <folder>
      • 2014-10-09 28205, 2014

      • ruaok_lappy starts builing
      • 2014-10-09 28211, 2014

      • nikki
        what are the dependencies? :/
      • 2014-10-09 28212, 2014

      • ianmcorvidae
        er
      • 2014-10-09 28227, 2014

      • kepstin-laptop
        nikki: they're in requirements.txt
      • 2014-10-09 28243, 2014

      • ianmcorvidae
        pip install -r <path to requirements.txt> should install them
      • 2014-10-09 28201, 2014

      • nikki
        ah
      • 2014-10-09 28206, 2014

      • ianmcorvidae
        I guess you have a VM anyway so might not be worth bothering with virtualenv anyway
      • 2014-10-09 28229, 2014

      • kepstin-laptop starts up an extractor built with debug info, and waits a couple minutes before taking a backtrace
      • 2014-10-09 28246, 2014

      • nikki
        too much python knowledge is required here :P
      • 2014-10-09 28245, 2014

      • Freso
        alastairp: I still haven't gotten set up. (Had to deal with food intake and submitting some data to MB too - I'm poking at setting up a venv now.)
      • 2014-10-09 28211, 2014

      • kepstin-laptop
        huh, looks like it's hanging in the peak detection code
      • 2014-10-09 28219, 2014

      • ianmcorvidae
        heh, that's a new error
      • 2014-10-09 28251, 2014

      • alastairp
        kepstin-laptop: cool! bug report please!
      • 2014-10-09 28204, 2014

      • alastairp
        I guess we;ve never had such a wide variety of audio before
      • 2014-10-09 28210, 2014

      • alastairp
        ianmcorvidae: thanks. I'll pass it on
      • 2014-10-09 28210, 2014

      • ianmcorvidae
        alastairp: where should bug reports go?
      • 2014-10-09 28220, 2014

      • alastairp
        github/mtg/essentia
      • 2014-10-09 28232, 2014

      • ianmcorvidae has another failure, this time breaking the entire abzsubmit process since it fails to write a JSON file :)
      • 2014-10-09 28241, 2014

      • Freso
        alastairp: "Edge case generators"... :)
      • 2014-10-09 28213, 2014

      • alastairp
        ianmcorvidae: yeah, as I said about an hour ago, it's a little flakey
      • 2014-10-09 28216, 2014

      • nikki
        :o
      • 2014-10-09 28218, 2014

      • nikki
        stuff might be happening
      • 2014-10-09 28219, 2014

      • alastairp
        I want to clean up that too
      • 2014-10-09 28222, 2014

      • alastairp
        nikki: awesome!
      • 2014-10-09 28230, 2014

      • Freso
        So. I guess I'm supposed to first build essentia, and then py setup install the client?
      • 2014-10-09 28247, 2014

      • alastairp
        Freso: no setup.py
      • 2014-10-09 28251, 2014

      • alastairp
        just run it from a checkout
      • 2014-10-09 28214, 2014

      • Freso
        Okay.
      • 2014-10-09 28221, 2014

      • nikki
        it said it wrote stuff to a temporary file which no longer exists, does that mean it submitted it?
      • 2014-10-09 28227, 2014

      • alastairp
        nikki: yep
      • 2014-10-09 28236, 2014

      • nikki
        how can I see stuff I submit? :P