#metabrainz

/

      • alastairp
        This is what I changed it to
      • 2020-08-24 23727, 2020

      • alastairp
        see line 26, I added eval_job["options"] to the call to evaluate_gaia
      • 2020-08-24 23747, 2020

      • alastairp
        and in the evaluate_gaia function I added the options arg, and passed the data to train_model
      • 2020-08-24 23717, 2020

      • alastairp
        now I'm going to run the tests again to make sure that nothing major is wrong
      • 2020-08-24 23702, 2020

      • jmp_music_
        by running `./test.sh`, right?
      • 2020-08-24 23705, 2020

      • alastairp
        yep
      • 2020-08-24 23715, 2020

      • alastairp
        tests passed. Now `git status` shows this:
      • 2020-08-24 23748, 2020

      • alastairp
      • 2020-08-24 23713, 2020

      • jmp_music_
        hmm ok!
      • 2020-08-24 23726, 2020

      • alastairp
        so this is correct. We're still in the middle of the merge. The "Changes to be committed" are your changes that applied correctly. The "Unmerged paths" is the file that we just fixed
      • 2020-08-24 23744, 2020

      • alastairp
        so now I'll do `git add dataset_eval/evaluate.py`, and `git commit`
      • 2020-08-24 23704, 2020

      • alastairp
        see that the commit message is something like this:
      • 2020-08-24 23707, 2020

      • alastairp
      • 2020-08-24 23716, 2020

      • alastairp
        this is exactly what we want
      • 2020-08-24 23736, 2020

      • alastairp
        now when I do `git log`, see the output:
      • 2020-08-24 23739, 2020

      • alastairp
      • 2020-08-24 23740, 2020

      • Gore joined the channel
      • 2020-08-24 23702, 2020

      • jmp_music_
        ok! I understand
      • 2020-08-24 23703, 2020

      • alastairp
        see that the second line is "Merge: [hash] [hash]" ? this indicates to us that it's a proper merge commit. Your merge didn't have this
      • 2020-08-24 23722, 2020

      • jmp_music_
        that's why it showed 1 parent
      • 2020-08-24 23747, 2020

      • alastairp
        yeah
      • 2020-08-24 23717, 2020

      • alastairp
        so now we have 4 more commits in your PR to deal with
      • 2020-08-24 23726, 2020

      • alastairp
        https://github.com/metabrainz/acousticbrainz-serv… this one is no longer needed, as we fixed it during the merge
      • 2020-08-24 23722, 2020

      • alastairp
      • 2020-08-24 23702, 2020

      • alastairp
        The easiest way to apply these commits would be to use `git cherry-pick`. This allows you to take a commit and apply it again as a new commit
      • 2020-08-24 23747, 2020

      • jmp_music_
        so the git can follow the steps of the commits as were before
      • 2020-08-24 23710, 2020

      • alastairp
        so `git cherry-pick 5c568aeb; git cherry-pick 36eb75ec` (note that you don't need to use the full commit hash, as long as it's unique it's fine, and 8-10 chars is almost always unique)
      • 2020-08-24 23726, 2020

      • alastairp
        right. note that you might see this if you run git status:
      • 2020-08-24 23738, 2020

      • alastairp
        Your branch and 'tzamalisp/ml_sklearn_structure' have diverged,
      • 2020-08-24 23738, 2020

      • alastairp
        and have 23 and 5 different commits each, respectively.
      • 2020-08-24 23712, 2020

      • jmp_music_
        ok. Would be a problem this one?
      • 2020-08-24 23717, 2020

      • alastairp
        this is because we've changed the history. we're going to have to use `git push --force` to tell github to replace this branch, because now they're not compatible with each other
      • 2020-08-24 23718, 2020

      • pristine___
        ishaanshah: how can I associate a mode with a component?
      • 2020-08-24 23734, 2020

      • jmp_music_
        ok
      • 2020-08-24 23754, 2020

      • jmp_music_
        I'll try to do this myself. You didn't push it right?
      • 2020-08-24 23707, 2020

      • alastairp
        correct, I didn't push it
      • 2020-08-24 23719, 2020

      • alastairp
      • 2020-08-24 23736, 2020

      • alastairp
        I believe it's not needed, because we already changed this code when we fixed the conflict
      • 2020-08-24 23742, 2020

      • alastairp
        I would skip that one
      • 2020-08-24 23749, 2020

      • alastairp
        try these steps and see how you go
      • 2020-08-24 23756, 2020

      • alastairp
        let me know if you have any questions
      • 2020-08-24 23758, 2020

      • ishaanshah
        pristine___: pass it through the props
      • 2020-08-24 23705, 2020

      • jmp_music_
        I added this function
      • 2020-08-24 23709, 2020

      • jmp_music_
        in that commit
      • 2020-08-24 23726, 2020

      • alastairp
        but `def evaluate_sklearn(eval_location, groundtruth_path, filelist_path, storage_dir, eval_job):` already existed
      • 2020-08-24 23731, 2020

      • alastairp
        with an empty block
      • 2020-08-24 23736, 2020

      • jmp_music_
      • 2020-08-24 23739, 2020

      • ishaanshah
      • 2020-08-24 23746, 2020

      • ishaanshah
        over here, add mode: "recs"
      • 2020-08-24 23708, 2020

      • alastairp
        right. I recommend that we leave this change for now
      • 2020-08-24 23720, 2020

      • jmp_music_
        ok!
      • 2020-08-24 23722, 2020

      • alastairp
        if you do what we discussed, and force push your branch, I'll merge it to master
      • 2020-08-24 23730, 2020

      • jmp_music_
        great
      • 2020-08-24 23735, 2020

      • alastairp
        and then you can make a new branch to apply all of the changes that you've made since then
      • 2020-08-24 23736, 2020

      • jmp_music_
        Can I ask you something?
      • 2020-08-24 23739, 2020

      • alastairp
        of course
      • 2020-08-24 23711, 2020

      • jmp_music_
        after building the image for running python 3
      • 2020-08-24 23720, 2020

      • jmp_music_
        and I started the AB server
      • 2020-08-24 23744, 2020

      • jmp_music_
        this error occurred on loading the sklearn tool
      • 2020-08-24 23753, 2020

      • jmp_music_
      • 2020-08-24 23720, 2020

      • alastairp
        mmmm, right
      • 2020-08-24 23731, 2020

      • jmp_music_
        it seems that it doesn't recognize the sklearn_evaluator container we built
      • 2020-08-24 23733, 2020

      • alastairp
        of course, because we're trying to run the same code in both python 2 and python 3
      • 2020-08-24 23754, 2020

      • alastairp
        correct, look at the left side - this is the dataset_evaluator and hl_extractor docker containers, which are running python 2
      • 2020-08-24 23743, 2020

      • alastairp
        OK, what we're going to do is a super hacky thing to only import the sklearn code in python 3
      • 2020-08-24 23700, 2020

      • jmp_music_
        we have to assign the sklearn tool scripts to the `dataset_evaluator_sklearn`?
      • 2020-08-24 23738, 2020

      • kori has quit
      • 2020-08-24 23741, 2020

      • jmp_music_
        with the `ADD` command inside the Dockefile.py3?
      • 2020-08-24 23741, 2020

      • alastairp
        the problem is because dataset_eval.evaluate imports the sklearn code, which requires pandas
      • 2020-08-24 23747, 2020

      • jmp_music_
        right
      • 2020-08-24 23700, 2020

      • jmp_music_
        and other libraries later on
      • 2020-08-24 23709, 2020

      • jmp_music_
        like joblib, etc.
      • 2020-08-24 23709, 2020

      • alastairp
        but this code is also used by the gaia evaluator
      • 2020-08-24 23734, 2020

      • alastairp
        what we can do is set an environment variable in the sklearn container, like this: https://github.com/metabrainz/acousticbrainz-serv…
      • 2020-08-24 23759, 2020

      • alastairp
        call it something like `MODEL_TRAINING_SKLEARN: 1`
      • 2020-08-24 23732, 2020

      • alastairp
      • 2020-08-24 23715, 2020

      • alastairp
        we can use `is_sklearn = os.getenv("MODEL_TRAINING_SKLEARN")` and `if is_sklearn == "1": import models.sklean....`
      • 2020-08-24 23740, 2020

      • alastairp
        so that it'll only perform this import in the python 3 container
      • 2020-08-24 23746, 2020

      • kori joined the channel
      • 2020-08-24 23716, 2020

      • jmp_music_
        ok! I see this kind of code for first time in my life! A great python update is inserted into my head right now 😄
      • 2020-08-24 23722, 2020

      • jmp_music_
        haha
      • 2020-08-24 23746, 2020

      • iliekcomputers
        shivam-kapila: hey, how goes the feedback export?
      • 2020-08-24 23730, 2020

      • jmp_music_
        alastairp I'm going to work with all these stuff and I'll keep you updated
      • 2020-08-24 23738, 2020

      • jmp_music_
        with the progress
      • 2020-08-24 23753, 2020

      • alastairp
        yeah, python is great that you don't have to import at the top of a file
      • 2020-08-24 23711, 2020

      • alastairp
        remember that when you import a file in python, it will execute the file
      • 2020-08-24 23730, 2020

      • alastairp
        so imagine if you had `print("x")` in the file, it would show that when you import it
      • 2020-08-24 23747, 2020

      • alastairp
        but if you have the print inside a function, or an if statement that evaluates to false, it wouldn't do it
      • 2020-08-24 23707, 2020

      • shivam-kapila
        iliekcomputers: waiting for interview. Will get to you
      • 2020-08-24 23710, 2020

      • alastairp
        this is why we use `if __name__ == "__main__"` to make things only happen when you run `python file.py` and not when you import it
      • 2020-08-24 23750, 2020

      • jmp_music_
        hmm, ok! Thanks for your guidance Alastair!
      • 2020-08-24 23738, 2020

      • Cyna[m]
        reosarevok: I don't see the plugins for postgres in the MB repo
      • 2020-08-24 23758, 2020

      • alastairp
        Cyna[m]: they no longer exist
      • 2020-08-24 23723, 2020

      • Cyna[m]
        So we dont need them anymore ?
      • 2020-08-24 23704, 2020

      • alastairp
        correct, with an updated version of postgres they are no longer needed
      • 2020-08-24 23718, 2020

      • MajorLurker has quit
      • 2020-08-24 23724, 2020

      • MajorLurker joined the channel
      • 2020-08-24 23731, 2020

      • MajorLurker has quit
      • 2020-08-24 23744, 2020

      • MajorLurker joined the channel
      • 2020-08-24 23732, 2020

      • HorusHorrendus
        alastairp: hi :) latest commit to my PR for python-musicbrainzng should fix the test under Python 2.7. Even though Python 2.7 is anyways already at EOL :)
      • 2020-08-24 23714, 2020

      • HorusHorrendus
        Is discussion about repos that are metabrainz related but not in the metabrainz Git org right in this channel? Probably no strict rules on such things or? :)
      • 2020-08-24 23716, 2020

      • alastairp
        this channel is fine for discussion
      • 2020-08-24 23748, 2020

      • alastairp
        I'm 100% committed to keeping python 2.7 support working for at least a few more years. EOL doesn't mean that people aren't using it :)
      • 2020-08-24 23752, 2020

      • alastairp
        HorusHorrendus: so is there still a missing test from my initial request?
      • 2020-08-24 23710, 2020

      • alastairp
        "we correctly send the auth header if one of these includes is set and the user has authenticated"
      • 2020-08-24 23725, 2020

      • HorusHorrendus
        Well there is no component test but if I recall correctly now all these things should be tested via unit tests
      • 2020-08-24 23720, 2020

      • HorusHorrendus
        not via Auth header (didnt really have time to dig into http digest auth, I think it doesnt set a header at first but is somehow used with redirects or so) but via checking "if auth is set to AUTH_YES, a digesthandler is added"
      • 2020-08-24 23742, 2020

      • alastairp
        OK, that's good enough for me
      • 2020-08-24 23700, 2020

      • alastairp
        what about AUTH_IFSET?
      • 2020-08-24 23709, 2020

      • HorusHorrendus
        and another test checks that AUTH_YES is set if user-genres is set
      • 2020-08-24 23713, 2020

      • HorusHorrendus
        AUTH_IFSET is not really tested ... could maybe add quickly one in the evening
      • 2020-08-24 23704, 2020

      • pristine___
        ishaanshah: I thought of updating the listens.listened_at value in here
      • 2020-08-24 23706, 2020

      • alastairp
        I'm just looking at the code flow for ifset, it seems to be if it's a collection request, but without an ID? I'm trying to work out what that means
      • 2020-08-24 23708, 2020

      • pristine___
      • 2020-08-24 23702, 2020

      • ksynwa has quit
      • 2020-08-24 23752, 2020

      • reosarevok
        Cyna[m]: remind me, do you have more edit conversions in the pipeline or should I work on the remaining ones? :)
      • 2020-08-24 23757, 2020

      • reosarevok
        (since you're doing some forms now)
      • 2020-08-24 23712, 2020

      • Cyna[m]
        Nah I dont have more edits
      • 2020-08-24 23723, 2020

      • Cyna[m]
        Only have the forms.
      • 2020-08-24 23703, 2020

      • shivam-kapila
        pristine___: how are you planning to do that
      • 2020-08-24 23735, 2020

      • shivam-kapila
        iliekcomputers: hi. My interview is postponed to an hour from now
      • 2020-08-24 23758, 2020

      • shivam-kapila
        I am here if you waana chat about something
      • 2020-08-24 23700, 2020

      • pristine___
        Idk atm :p have just thought of updating listen.listened_at here
      • 2020-08-24 23739, 2020

      • _lucifer
        alastairp: are you familiar with essentia's AudioLoader code?
      • 2020-08-24 23709, 2020

      • _lucifer
        *AudioLoader algorithm
      • 2020-08-24 23737, 2020

      • shivam-kapila
        pristine___: why would you update in componentwillunmount
      • 2020-08-24 23712, 2020

      • shivam-kapila
        Thats used for actions when a component detaches
      • 2020-08-24 23714, 2020

      • alastairp
        _lucifer: what specifically? I don't know about how it works internally
      • 2020-08-24 23734, 2020

      • shivam-kapila
      • 2020-08-24 23741, 2020

      • _lucifer
        alastairp: some time ago when I was trying to get streaming_extractor_music to work with a file descriptor or a stream instead of a path, dmitry had told me that this was not currently possible due to the algorithm depending on AudioLoader. And that it was a goal to separate the components. i wanted to work on that but I do not know where and how to start
      • 2020-08-24 23749, 2020

      • alastairp
        right. does your stream include an encoded music format, or is it raw PCM?
      • 2020-08-24 23707, 2020

      • pristine___
        shivam-kapila: oh right. I thought it is called just before the component is rendered so that my be a good place to update. Thanks
      • 2020-08-24 23705, 2020

      • shivam-kapila
        Thats componentWillMount
      • 2020-08-24 23736, 2020

      • shivam-kapila
        Sorry componentdidmount
      • 2020-08-24 23752, 2020

      • _lucifer
        I am not sure about that alastairp
      • 2020-08-24 23755, 2020

      • shivam-kapila
        Componentwillmount is deprecated
      • 2020-08-24 23716, 2020

      • shivam-kapila
        pristine___: can you tell me what your end goal is?
      • 2020-08-24 23749, 2020

      • alastairp
        _lucifer: My guess is that we could have a separate algorithm that uses ffmpeg functions to open a stream instead of a file: https://github.com/MTG/essentia/blob/ba79be6515f2…
      • 2020-08-24 23717, 2020

      • alastairp
        but that's the level of my understanding. I don't even know if ffmpeg has a function to load from an open file handle or a data stream
      • 2020-08-24 23737, 2020

      • _lucifer
        it can open file handles not sure about data stream
      • 2020-08-24 23710, 2020

      • _lucifer
        but chromaprint supports data stream and it uses ffmpeg underneath so i believe it should be possible
      • 2020-08-24 23734, 2020

      • alastairp
      • 2020-08-24 23738, 2020

      • alastairp
        and the avio_reading.c example
      • 2020-08-24 23701, 2020

      • pristine___
      • 2020-08-24 23734, 2020

      • alastairp
        you could look at the chromaprint load functions to see how it calls ffmpeg too
      • 2020-08-24 23729, 2020

      • _lucifer
        Yeah, makes sense. so i'll try to create a new algorithm for working with audiostream
      • 2020-08-24 23759, 2020

      • _lucifer
        also, you had mentioned that streaming_extractor_music processes the stream multiple times
      • 2020-08-24 23710, 2020

      • _lucifer
        how can that be avoided
      • 2020-08-24 23725, 2020

      • pristine___
        Recs don't have listened_at and because of this unix epoch is being rendered. I just want to update the listens to remove listened_at in listens which are being passed to Brainzplayer. ishaanshah gave me the suggestion to use a separate mode for that and conditionally handle listened_at there.
      • 2020-08-24 23728, 2020

      • pristine___
        shivam-kapila: