#metabrainz

/

      • kartikgupta0909
        yes
      • 2016-05-05 12606, 2016

      • alastairp
        this is how we currently do it for data training
      • 2016-05-05 12618, 2016

      • alastairp
        we extract files from the database to a directory in /tmp
      • 2016-05-05 12619, 2016

      • kartikgupta0909
        and I think we should also check if the client machine has that much of space
      • 2016-05-05 12631, 2016

      • alastairp
        yes, good point!
      • 2016-05-05 12642, 2016

      • alastairp
        so the server can respond in the status message how big the file is
      • 2016-05-05 12649, 2016

      • kartikgupta0909
        yes
      • 2016-05-05 12600, 2016

      • kartikgupta0909
        in case the client machine cant handle that the user will be informed
      • 2016-05-05 12603, 2016

      • alastairp
        and the client can either download it, or send a message "sorry, I don't have enough space, please cancel this job"
      • 2016-05-05 12610, 2016

      • alastairp
        yes
      • 2016-05-05 12616, 2016

      • kartikgupta0909
        yeah
      • 2016-05-05 12642, 2016

      • kartikgupta0909
        so the main concern is the format of storing and rendering of the data from the database to files
      • 2016-05-05 12608, 2016

      • alastairp
        I'm not sure that will be a very big problem
      • 2016-05-05 12638, 2016

      • alastairp
        like I suggested for many parts - let's start simple and when we have something working we can optimise it
      • 2016-05-05 12643, 2016

      • kartikgupta0909
        no you were saying that since the data might be too large it might take very long
      • 2016-05-05 12646, 2016

      • kartikgupta0909
        yes
      • 2016-05-05 12657, 2016

      • alastairp
        for example. We could start by using json and compressing it with bzip2
      • 2016-05-05 12615, 2016

      • alastairp
        right - the process of retrieving the data from the database
      • 2016-05-05 12637, 2016

      • alastairp
        for example, I recently made a model with 13000 items, it took about 10 minutes to load the files from the database
      • 2016-05-05 12613, 2016

      • kartikgupta0909
        can you tell me how can I have a look at the frame level data. I ll try to think something, I like to work on these kind of problems
      • 2016-05-05 12633, 2016

      • alastairp
        do you have the commandline submission tool?
      • 2016-05-05 12638, 2016

      • kartikgupta0909
        yes
      • 2016-05-05 12659, 2016

      • kartikgupta0909
        I have the output for the extractor but it doesnt contain the frame level data i think
      • 2016-05-05 12602, 2016

      • alastairp
        nope
      • 2016-05-05 12626, 2016

      • alastairp
        create a file like this:
      • 2016-05-05 12634, 2016

      • alastairp
      • 2016-05-05 12646, 2016

      • alastairp
        and then run
      • 2016-05-05 12659, 2016

      • alastairp
        ./streaming_extractor_music <mp3> <output.json> <thatfile>
      • 2016-05-05 12630, 2016

      • alastairp
        that will make a frame-level file
      • 2016-05-05 12627, 2016

      • kartikgupta0909
        its running now
      • 2016-05-05 12629, 2016

      • alastairp
        OK, I think this is probably enough for now. I don't want to go into too much detail in the later parts of the project, because I want the option to change our mind about things depending on how this part goes
      • 2016-05-05 12639, 2016

      • kartikgupta0909
        yeah sure
      • 2016-05-05 12644, 2016

      • kartikgupta0909
        so for now
      • 2016-05-05 12648, 2016

      • alastairp
        you should create tickets in jira for everything that we've talked about
      • 2016-05-05 12650, 2016

      • kartikgupta0909
        I will work on making the PPA
      • 2016-05-05 12656, 2016

      • kartikgupta0909
        yes
      • 2016-05-05 12611, 2016

      • alastairp
        the size of a ticket shouldn't be more than a few days work (take a guess at how much work you can do in 2 or 3 days)
      • 2016-05-05 12633, 2016

      • kartikgupta0909
        oh okay.
      • 2016-05-05 12637, 2016

      • alastairp
        don't worry if things take longer or shorter than you expect, we can refine our estimates as we go
      • 2016-05-05 12648, 2016

      • alastairp
        the idea is that we don't want a single ticket that says "do all the work"
      • 2016-05-05 12619, 2016

      • alastairp
        because then there isn't any point in having the ticket, since it's difficult to see what the task entails, and where you are in the ticket
      • 2016-05-05 12640, 2016

      • kartikgupta0909
        yeah, I ll keep that in my mind
      • 2016-05-05 12645, 2016

      • alastairp
        on the other hand, small tickets can be understood easily, discussed, moved around if needed
      • 2016-05-05 12657, 2016

      • kartikgupta0909
        yes
      • 2016-05-05 12600, 2016

      • alastairp
        and also, it's really satisfying to close lots of tickets :)
      • 2016-05-05 12613, 2016

      • kartikgupta0909
        yes, plus it ll look good on my github too :p
      • 2016-05-05 12605, 2016

      • alastairp
        ok 👍 great!
      • 2016-05-05 12627, 2016

      • kartikgupta0909
        thanks for your time today :)
      • 2016-05-05 12643, 2016

      • Gentlecat
      • 2016-05-05 12646, 2016

      • Gentlecat
        maybe we should move this to readthedocs too https://acousticbrainz.org/data
      • 2016-05-05 12627, 2016

      • alastairp
        yeah, I think you said that
      • 2016-05-05 12641, 2016

      • alastairp
        right, that page could be a link to RTD for the API
      • 2016-05-05 12602, 2016

      • alastairp
        then link to other things (examples - view page, data & datasets - dataset page)
      • 2016-05-05 12619, 2016

      • MBJenkins
        Project acousticbrainz-server build #53: SUCCESS in 4 min 45 sec: https://ci.metabrainz.org/job/acousticbrainz-serv…
      • 2016-05-05 12653, 2016

      • JonnyJD has quit
      • 2016-05-05 12641, 2016

      • rahulr joined the channel
      • 2016-05-05 12610, 2016

      • Jormangeud joined the channel
      • 2016-05-05 12637, 2016

      • MBJenkins
        Project acousticbrainz-server build #54: SUCCESS in 4 min 47 sec: https://ci.metabrainz.org/job/acousticbrainz-serv…
      • 2016-05-05 12608, 2016

      • regagain has quit
      • 2016-05-05 12630, 2016

      • UmkaDK has quit
      • 2016-05-05 12627, 2016

      • UmkaDK joined the channel
      • 2016-05-05 12639, 2016

      • kanha joined the channel
      • 2016-05-05 12609, 2016

      • regagain joined the channel
      • 2016-05-05 12648, 2016

      • Gentlecat
        alastairp: Dan is here btw, if you and Dmitry want to talk about the project
      • 2016-05-05 12649, 2016

      • rahulr has quit
      • 2016-05-05 12645, 2016

      • rahulr joined the channel
      • 2016-05-05 12618, 2016

      • MajorLurker has quit
      • 2016-05-05 12650, 2016

      • mihaitish joined the channel
      • 2016-05-05 12659, 2016

      • alastairp
        Gentlecat: ah, let me see if he's here
      • 2016-05-05 12607, 2016

      • MBJenkins has quit
      • 2016-05-05 12636, 2016

      • MBJenkins joined the channel
      • 2016-05-05 12614, 2016

      • mihaitish has quit
      • 2016-05-05 12616, 2016

      • mihaitish joined the channel
      • 2016-05-05 12650, 2016

      • mihaitish has quit
      • 2016-05-05 12641, 2016

      • Freso
        alastairp: Can you send me a list of the AB specific points of the HTTPS blog to freso@meb?
      • 2016-05-05 12636, 2016

      • Freso
        I'm babysitting my baby cousin today, so not sure if I'll actually manage to write it up (she's currently asleep, but I'd rather not be in the middle of the blog post when she wakes up, so not sure I'll start properly on it today).
      • 2016-05-05 12616, 2016

      • Gentlecat
        alastairp: I left already, it got too noisy with that concert or whatever it is
      • 2016-05-05 12625, 2016

      • Gentlecat
        I guess you can still talk to him if he's there
      • 2016-05-05 12628, 2016

      • alastairp
        Gentlecat: concert finished :)
      • 2016-05-05 12632, 2016

      • alastairp
        yeah, we spoke briefly
      • 2016-05-05 12640, 2016

      • alastairp
        let's plan something for the beginning of next week
      • 2016-05-05 12637, 2016

      • Gentlecat
        yeah, good idea
      • 2016-05-05 12614, 2016

      • Gentlecat
        he wanted to work on http://tickets.musicbrainz.org/browse/AB-71 to get familiar with the editor
      • 2016-05-05 12619, 2016

      • mihaitish joined the channel
      • 2016-05-05 12635, 2016

      • Gentlecat
        I showed him how to set up oauth and how JS builds work
      • 2016-05-05 12609, 2016

      • alastairp
        yeah, he said
      • 2016-05-05 12631, 2016

      • alastairp
        I suggested the ideas we had about messybrainz, etc
      • 2016-05-05 12642, 2016

      • alastairp
        and said that we could modify the project to either be 1) do messybrainz and then modify dataset to use that, or 2) change datasets to have non-mbids when needed, with an API to create them and then retrofit messybrainz in if there's time
      • 2016-05-05 12649, 2016

      • Freso
        Gentlecat: Him = "Dan" = your GSoC student?
      • 2016-05-05 12653, 2016

      • Freso
        Or?
      • 2016-05-05 12654, 2016

      • alastairp
        yes
      • 2016-05-05 12658, 2016

      • Freso
        Cool. :)
      • 2016-05-05 12604, 2016

      • alastairp
        after thinking about it I don't think there's much difference in which order we do
      • 2016-05-05 12604, 2016

      • Gentlecat
        yes, pretty cool
      • 2016-05-05 12627, 2016

      • alastairp
        Freso: we'll be doing a significant amount of in-person meeting, but will make sure that we publicise the results
      • 2016-05-05 12637, 2016

      • Freso
        Gentlecat: I know it's easy to talk things over in person, but it'd be great if you'd leave a trail of conver... great, thanks alastairp :D
      • 2016-05-05 12641, 2016

      • alastairp
        Freso: OK, will email you by this afternoon
      • 2016-05-05 12601, 2016

      • Gentlecat
        yeah, we'll try to
      • 2016-05-05 12608, 2016

      • alastairp
        I think the good idea will be to do what I started with kartikgupta0909 this morning
      • 2016-05-05 12617, 2016

      • alastairp
        we'll have an AB component in jira for the project
      • 2016-05-05 12624, 2016

      • alastairp
        results of discussions can turn into tickets
      • 2016-05-05 12636, 2016

      • alastairp
        then discussion with everyone else can happen there
      • 2016-05-05 12610, 2016

      • Gentlecat
        for my own work on gsoc I was using versions
      • 2016-05-05 12621, 2016

      • Gentlecat
      • 2016-05-05 12633, 2016

      • alastairp
        geoh yeah
      • 2016-05-05 12637, 2016

      • alastairp
        I guess we could
      • 2016-05-05 12638, 2016

      • Gentlecat
        it's easy to track the progress that way and see how much is yet to be done
      • 2016-05-05 12654, 2016

      • alastairp
        Gentlecat: what did we decide on for AB https/login?
      • 2016-05-05 12659, 2016

      • alastairp
        redirect the login page to https
      • 2016-05-05 12606, 2016

      • alastairp
        change oauth redirect
      • 2016-05-05 12613, 2016

      • Gentlecat
        yes, that's the minimum for now
      • 2016-05-05 12620, 2016

      • Gentlecat
        to make sure that people can log in
      • 2016-05-05 12630, 2016

      • alastairp
        the thing is, if they go to dataset in http, they need to be on https
      • 2016-05-05 12644, 2016

      • alastairp
        perhaps we redirect all website now. leave api
      • 2016-05-05 12650, 2016

      • alastairp
        I think that makes more sense
      • 2016-05-05 12652, 2016

      • Gentlecat
        not sure what you mean
      • 2016-05-05 12606, 2016

      • alastairp
        is a session cookie shared over http/https?
      • 2016-05-05 12614, 2016

      • Gentlecat
        but yeah, redirecting everything apart from api (old one too) seems sensible
      • 2016-05-05 12621, 2016

      • Gentlecat
        I think it is
      • 2016-05-05 12625, 2016

      • alastairp
        ahhh
      • 2016-05-05 12629, 2016

      • alastairp
        that's kind of buggy
      • 2016-05-05 12633, 2016

      • alastairp
        OK. hmm
      • 2016-05-05 12647, 2016

      • Gentlecat
        it's attached to a host, iirc
      • 2016-05-05 12648, 2016

      • alastairp
        actually, only the old API needs to stay on http
      • 2016-05-05 12656, 2016

      • Gentlecat
        right
      • 2016-05-05 12658, 2016

      • alastairp
        new api can be https-only
      • 2016-05-05 12608, 2016

      • alastairp
        and really, the only important one is the POST
      • 2016-05-05 12641, 2016

      • alastairp
        ok. let's say that website redirects, oldapi doesn't
      • 2016-05-05 12645, 2016

      • alastairp
        but we won't say it's oldapi
      • 2016-05-05 12649, 2016

      • Freso
        alastairp: FWIW, mb.o will be all-HTTPS, except for /ws/.
      • 2016-05-05 12601, 2016

      • Freso
        So in that regard, it might make sense to do the same for AB.
      • 2016-05-05 12603, 2016

      • alastairp
        Freso: right. that fits in well
      • 2016-05-05 12622, 2016

      • alastairp
        Gentlecat: when we announce the new API/deprecate the old one we say that the new one can only be used over https
      • 2016-05-05 12639, 2016

      • alastairp
        then when we turn off the old one, everything will be on https
      • 2016-05-05 12658, 2016

      • Zastai joined the channel
      • 2016-05-05 12613, 2016

      • Slurpee joined the channel
      • 2016-05-05 12613, 2016

      • Slurpee has quit
      • 2016-05-05 12613, 2016

      • Slurpee joined the channel
      • 2016-05-05 12603, 2016

      • Zastai has quit
      • 2016-05-05 12648, 2016

      • kartikgupta0909 has quit
      • 2016-05-05 12657, 2016

      • D4RK-PH0ENiX has quit
      • 2016-05-05 12620, 2016

      • mike_aiir joined the channel
      • 2016-05-05 12624, 2016

      • Zastai joined the channel
      • 2016-05-05 12612, 2016

      • mike_aiir
        Hey guys, are you having some problems with your api service? Our current music solution which still uses your web service seems to reporting 503 errors intermittently
      • 2016-05-05 12618, 2016

      • diana_olhovyk has quit
      • 2016-05-05 12608, 2016

      • Leo_Verto
        zas, ^
      • 2016-05-05 12611, 2016

      • D4RK-PH0ENiX joined the channel
      • 2016-05-05 12641, 2016

      • kepstin
        mike_aiir: due to bandwidth limits with the current musicbrainz hosting, rather strict rate limits are currently being applied to overall api requests. This is hopefully gonna be a temporary thing until the hosting situation is sorted out.
      • 2016-05-05 12600, 2016

      • mike_aiir
        @kepstin okay cool, any ETA?
      • 2016-05-05 12628, 2016

      • kepstin
        mike_aiir: not that i'm a party to :/