#metabrainz

/

      • hellska_
        let me read it carefully, I'm actually reading now!
      • 2016-07-04 18627, 2016

      • hellska_
        alastairp: yes I think there's enough to start working on! I think it worth to make a specific plan for this part, as gentlecat suggested in the midterm feedback having a good plan is a very good thing :)
      • 2016-07-04 18659, 2016

      • Gentlecat
        we already had it iirc though
      • 2016-07-04 18630, 2016

      • Gentlecat
        and you started working on implementing non-mbid submissions
      • 2016-07-04 18645, 2016

      • hellska_
        yes, I was thinking that I need a plan to integrate MessyBrainz (that I didn't checked yet!)
      • 2016-07-04 18654, 2016

      • hellska_
        so basically the idea is to call MessyBrainz to generate the IDs while the rest of the project remains the same right?!
      • 2016-07-04 18610, 2016

      • Gentlecat
        alastairp: ^
      • 2016-07-04 18626, 2016

      • Gentlecat
        (I need to read it too, didn't have a chance yet)
      • 2016-07-04 18655, 2016

      • alastairp
        right, I didn't originally suggest to integrate MessyBrainz because I thought we could start basic, just generating our own IDs
      • 2016-07-04 18652, 2016

      • alastairp
        but after talking with ruaok he convinced me that adding MessyBrainz in now is a good idea - especially because I think it's a good idea to be able to make these identifiers as stable as soon as we can
      • 2016-07-04 18625, 2016

      • hellska_
        alastairp: agreed!
      • 2016-07-04 18630, 2016

      • alastairp
        I mean to say, if we generate our own ids now, then in a few months add messybrainz support in, we will have a handful of ids which are neither mbids nor messybrainz ids
      • 2016-07-04 18612, 2016

      • alastairp
        we could use our enum `gid_type` column to show this, but then we'd have to support 3 id types in acousticbrainz
      • 2016-07-04 18630, 2016

      • Gentlecat
        it shouldn't be hard to make them messybrainz ids though, right?
      • 2016-07-04 18641, 2016

      • alastairp
        we could manually import them into messybrainz
      • 2016-07-04 18648, 2016

      • Gentlecat
        right, that's what I mean
      • 2016-07-04 18650, 2016

      • alastairp
        but I don't want to do that
      • 2016-07-04 18602, 2016

      • alastairp
        e.g., what if this combination of artist/title has already been added to messybrainz?
      • 2016-07-04 18615, 2016

      • Gentlecat
        might be easier to just implement sending a query to messybrainz now
      • 2016-07-04 18619, 2016

      • alastairp
        right
      • 2016-07-04 18625, 2016

      • alastairp
        it's 1 http call :)
      • 2016-07-04 18635, 2016

      • alastairp
        zas: do we have an https certificate for messybrainz.org?
      • 2016-07-04 18637, 2016

      • alastairp
        on babar
      • 2016-07-04 18629, 2016

      • hellska_
        gentlecat: alastairp: I really have to look into messybrainz guys! But if I understand we will have 3 types that are MBID, messybrainzIDs and dataset items IDs (generated in messybrainz)
      • 2016-07-04 18603, 2016

      • alastairp
        nope, the last 2 are the same thing
      • 2016-07-04 18615, 2016

      • alastairp
        we'll have 2 types, mbid, messyid
      • 2016-07-04 18638, 2016

      • hellska_
        ok, so what's the third type you mentioned?
      • 2016-07-04 18651, 2016

      • alastairp
        oh, I meant a hypothetical situation
      • 2016-07-04 18618, 2016

      • alastairp
        where we would have mbids, our custom dataset ids, and then messyids when we add support for messybrainz
      • 2016-07-04 18637, 2016

      • alastairp
        but we should add support for messybrainz now, and the second format (custom dataset ids) will stop existing
      • 2016-07-04 18600, 2016

      • alastairp
        messybrainz has an api which takes {"artist": "artistname", "title": "tracktitle"} [and some extra metadata if you have it] and returns you a messybrainz id
      • 2016-07-04 18619, 2016

      • alastairp
        if the same text artist name and track title exist in the database, you will get the same id back
      • 2016-07-04 18638, 2016

      • hellska_
        what about datasets without artist/title?!
      • 2016-07-04 18602, 2016

      • hellska_
        we should not accept this items?!
      • 2016-07-04 18610, 2016

      • alastairp
        correct
      • 2016-07-04 18619, 2016

      • alastairp
        we need to set a limit somewhere
      • 2016-07-04 18630, 2016

      • alastairp
        our limit should be that we need at least those two items
      • 2016-07-04 18633, 2016

      • mildused joined the channel
      • 2016-07-04 18637, 2016

      • hellska_
        Ok! that's fine!
      • 2016-07-04 18657, 2016

      • alastairp
        this makes sense in the context of dataset building - we should know what is in our dataset!
      • 2016-07-04 18607, 2016

      • hellska_
        yes I know, I was just thinking to real datasets like the ballroom that has no artist and sometimes not even a title, but uses the filename :(
      • 2016-07-04 18618, 2016

      • alastairp
        hrm
      • 2016-07-04 18634, 2016

      • hellska_
        anyway we can set the stadard behaviour and then try to find a way to integrate more complex situations
      • 2016-07-04 18651, 2016

      • alastairp
        I guess that makes sense, You don't need to know who it's by, just what the style is
      • 2016-07-04 18607, 2016

      • Gentlecat
        I think that might be going too far
      • 2016-07-04 18645, 2016

      • Gentlecat
        how many datasets without artist and title are there? and how useful are they actually going to be?
      • 2016-07-04 18611, 2016

      • hellska_
        good point!
      • 2016-07-04 18641, 2016

      • Gentlecat
        this seems to be getting a bit too complex for a project we have here
      • 2016-07-04 18607, 2016

      • Gentlecat
        and we aren't even half-done with what was planned
      • 2016-07-04 18633, 2016

      • hellska_
        yeah you're right! I'll make a plan as simple as possible and I'll share with you so you can comment.
      • 2016-07-04 18623, 2016

      • Gentlecat
        maybe focus on implementing it too
      • 2016-07-04 18624, 2016

      • hellska_
        I'm just thinking in bullet point! Not a complex document ;)
      • 2016-07-04 18658, 2016

      • hellska_
        so then I can start working on the code! For the schema change I already can start to change the code!
      • 2016-07-04 18603, 2016

      • blozo joined the channel
      • 2016-07-04 18651, 2016

      • blozo has quit
      • 2016-07-04 18651, 2016

      • Mineo joined the channel
      • 2016-07-04 18652, 2016

      • yeeeargh joined the channel
      • 2016-07-04 18618, 2016

      • Zialus has quit
      • 2016-07-04 18621, 2016

      • Zialus joined the channel
      • 2016-07-04 18623, 2016

      • kartikgupta0909 has quit
      • 2016-07-04 18615, 2016

      • alastairp
        me and dima and hellska_ just had a chat about this
      • 2016-07-04 18632, 2016

      • alastairp
        hellska_ will add a comment to the jira ticket about it
      • 2016-07-04 18601, 2016

      • alastairp
        but the general gist is that we'll only accept submissions of datasets with at least that metadata and use messybrainz
      • 2016-07-04 18650, 2016

      • ruaok
        Freso: dockerizing all of the services hosted at DWNI
      • 2016-07-04 18603, 2016

      • ruaok
        so that we can spin them up at NewHost.
      • 2016-07-04 18603, 2016

      • alastairp
        do beta and main have the same rate limiting?
      • 2016-07-04 18617, 2016

      • ruaok
      • 2016-07-04 18601, 2016

      • armalcolite
        ruaok: fixed init_test_db()
      • 2016-07-04 18650, 2016

      • MBJenkins
        Project listenbrainz-server build #83: STILL FAILING in 1 min 6 sec: https://ci.metabrainz.org/job/listenbrainz-server…
      • 2016-07-04 18623, 2016

      • Zastai joined the channel
      • 2016-07-04 18650, 2016

      • JesseW joined the channel
      • 2016-07-04 18651, 2016

      • Freso
        ruaok: Okay, so bitmap wasn't putting a number on what you asked him then, according to his own answer to my question. :)
      • 2016-07-04 18613, 2016

      • armalcolite
        ruaok: the server script needs to be changed. i think the sed command does not work any more. (https://www.irccloud.com/pastebin/ZRjtV5ya/)
      • 2016-07-04 18652, 2016

      • ruaok
        test stuff is confirmed working. very good.
      • 2016-07-04 18613, 2016

      • armalcolite
        oh. nice!
      • 2016-07-04 18640, 2016

      • ruaok
        which sed command?
      • 2016-07-04 18653, 2016

      • armalcolite
        line 14 in the above script
      • 2016-07-04 18609, 2016

      • Gentlecat
        alastairp: ok
      • 2016-07-04 18616, 2016

      • bitmap
        Freso: the context was just mbs though, since I was never working on dockerizing everything
      • 2016-07-04 18617, 2016

      • ruaok
        what script is that even?
      • 2016-07-04 18621, 2016

      • armalcolite
        i changed the config.py.sample so.
      • 2016-07-04 18638, 2016

      • armalcolite
        it is the script which runs on jenkins, Gentlecat shared it with me.
      • 2016-07-04 18650, 2016

      • ruaok
        ah. I see.
      • 2016-07-04 18619, 2016

      • ruaok
        write a small snipped in python that calls urlparse like in the code you just touched.
      • 2016-07-04 18626, 2016

      • github joined the channel
      • 2016-07-04 18626, 2016

      • github
        [listenbrainz-server] mayhem closed pull request #87: Cleanup kafka bits & Fix tests setup (master...fix_tests_cleanup) https://github.com/metabrainz/listenbrainz-server…
      • 2016-07-04 18626, 2016

      • github has left the channel
      • 2016-07-04 18628, 2016

      • armalcolite
        we require in bash?
      • 2016-07-04 18635, 2016

      • ruaok
        ok, merged. go ahead and rebase the api-compat branch and I'll check it again.
      • 2016-07-04 18645, 2016

      • ruaok
        not sure. Gentlecat?
      • 2016-07-04 18647, 2016

      • Leftmost
        Freso, no updates from me this week.
      • 2016-07-04 18655, 2016

      • Gentlecat
        ?
      • 2016-07-04 18658, 2016

      • armalcolite
        the script is bash, Line1 :P
      • 2016-07-04 18603, 2016

      • armalcolite
      • 2016-07-04 18608, 2016

      • ruaok
        me neither. last week was super calm and quiet.
      • 2016-07-04 18614, 2016

      • ruaok slinks out to go drink beer.
      • 2016-07-04 18631, 2016

      • MBJenkins
        Project listenbrainz-server build #84: STILL FAILING in 1 min 8 sec: https://ci.metabrainz.org/job/listenbrainz-server…
      • 2016-07-04 18642, 2016

      • armalcolite
        Gentlecat: ^^
      • 2016-07-04 18601, 2016

      • ruaok
        Gentlecat: see the sed line in the jenkins setup for listenbrainz. it can't parse the updated sqlalchemy connect strings.
      • 2016-07-04 18615, 2016

      • Gentlecat
        send me an updated version and I'll replace it
      • 2016-07-04 18600, 2016

      • armalcolite
        Gentlecat: i'll update it and share with you.
      • 2016-07-04 18649, 2016

      • armalcolite
        ruaok: rebased.
      • 2016-07-04 18631, 2016

      • MBJenkins
        Project listenbrainz-server build #85: STILL FAILING in 1 min 8 sec: https://ci.metabrainz.org/job/listenbrainz-server…
      • 2016-07-04 18651, 2016

      • ruaok
        armalcolite: I'm on 38482f8e7a0e and I still see references to kakfa and the server refusing to start.
      • 2016-07-04 18615, 2016

      • ruaok
      • 2016-07-04 18650, 2016

      • armalcolite
        oops, api_tools was created in this PR.
      • 2016-07-04 18604, 2016

      • armalcolite
        so it was not cleaned when i merged it. i'll do it asap.
      • 2016-07-04 18645, 2016

      • armalcolite
        ruaok: try, git pull
      • 2016-07-04 18650, 2016

      • MBJenkins
        Project listenbrainz-server build #86: STILL FAILING in 1 min 7 sec: https://ci.metabrainz.org/job/listenbrainz-server…
      • 2016-07-04 18604, 2016

      • ruaok
        armalcolite: before you push the next commmit, please do a "grep -ri kafka *" you missed another reference.
      • 2016-07-04 18618, 2016

      • ruaok
        and are you actually *running* this code before checking it in?
      • 2016-07-04 18647, 2016

      • armalcolite
        ruaok: actually i was working on jenkins server script
      • 2016-07-04 18600, 2016

      • ruaok
        ok, do one thing at a time please.
      • 2016-07-04 18602, 2016

      • armalcolite
        i used searchmonkey to find another one, changed it as well.
      • 2016-07-04 18634, 2016

      • MBJenkins
        Project listenbrainz-server build #87: STILL FAILING in 1 min 6 sec: https://ci.metabrainz.org/job/listenbrainz-server…
      • 2016-07-04 18648, 2016

      • armalcolite
        ruaok: and i have to stash "Dockerfile" before switching to another branch and then apply, which makes things bit complicated.
      • 2016-07-04 18607, 2016

      • ruaok
        why are you not running the docker setup?
      • 2016-07-04 18627, 2016

      • armalcolite
        it still downloads everytime i update a file.
      • 2016-07-04 18642, 2016

      • ruaok
        we need to fix that.
      • 2016-07-04 18645, 2016

      • armalcolite
        and that same ReadTimeout error while fetching package from pip
      • 2016-07-04 18655, 2016

      • ruaok
        you need to be working with the docker setup.
      • 2016-07-04 18601, 2016

      • armalcolite
        yeah.
      • 2016-07-04 18603, 2016

      • blozo joined the channel
      • 2016-07-04 18607, 2016

      • armalcolite
        Gentlecat: i think this can do the job: https://ptpb.pw/Pd0U.sh
      • 2016-07-04 18606, 2016

      • armalcolite
        ruaok: how did you cache the packages in docker?
      • 2016-07-04 18643, 2016

      • ruaok
        the dockerfiles need to be organized to not invalidate everything all the time.
      • 2016-07-04 18653, 2016

      • ruaok
        but I've done that and here they build quite fast.
      • 2016-07-04 18600, 2016

      • ruaok
        not sure what to do.
      • 2016-07-04 18606, 2016

      • ruaok
        try two things:
      • 2016-07-04 18616, 2016

      • ruaok
        1. a build in a clean checkout of the code
      • 2016-07-04 18641, 2016

      • ruaok
        2. send me the output of a complete run and then let me know which portions run everytime you do a build.
      • 2016-07-04 18609, 2016

      • blozo has quit
      • 2016-07-04 18614, 2016

      • ruaok
        lol, nice Rajoy Division image, reosarevok
      • 2016-07-04 18632, 2016

      • armalcolite
        clean code did not download a thing!
      • 2016-07-04 18610, 2016

      • armalcolite
        my earlier code was even with master. not sure why this is happening in this checkout.
      • 2016-07-04 18651, 2016

      • thebradleys joined the channel
      • 2016-07-04 18602, 2016

      • hellska_ has left the channel
      • 2016-07-04 18604, 2016

      • hellska joined the channel
      • 2016-07-04 18644, 2016

      • diana_olhovyk_ joined the channel
      • 2016-07-04 18613, 2016

      • ruaok
        ok, good to know. work with the clean code then. :)
      • 2016-07-04 18636, 2016

      • armalcolite
        i am starting to like docker now, since that cache thing is fixed. :D
      • 2016-07-04 18629, 2016

      • Leo_Verto
        Freso, I want out, spent the week working on non-opensource projects :O
      • 2016-07-04 18639, 2016

      • Leo_Verto
        at least not open source while the code is a huge mess :P
      • 2016-07-04 18659, 2016

      • armalcolite
        ruaok: i checked the master on docker and listens are going smoothly.
      • 2016-07-04 18610, 2016

      • Gentlecat
      • 2016-07-04 18621, 2016

      • ruaok
        k, I'll look at the meeting.
      • 2016-07-04 18642, 2016

      • armalcolite
        ruaok: i also tried with forcefully 400's and its fine.
      • 2016-07-04 18651, 2016

      • ruaok
        k
      • 2016-07-04 18654, 2016

      • armalcolite
        i think the issue was something else.
      • 2016-07-04 18657, 2016

      • MBJenkins
        Project listenbrainz-server build #88: STILL FAILING in 1 min 6 sec: https://ci.metabrainz.org/job/listenbrainz-server…