#metabrainz

/

      • D4RK-PH0ENiX joined the channel
      • 2019-03-01 06007, 2019

      • CallerNo6 joined the channel
      • 2019-03-01 06048, 2019

      • CallerNo6 has quit
      • 2019-03-01 06043, 2019

      • PelvicSorcery has quit
      • 2019-03-01 06015, 2019

      • Rotab joined the channel
      • 2019-03-01 06047, 2019

      • yokel has quit
      • 2019-03-01 06012, 2019

      • Slurpee joined the channel
      • 2019-03-01 06012, 2019

      • Slurpee has quit
      • 2019-03-01 06012, 2019

      • Slurpee joined the channel
      • 2019-03-01 06025, 2019

      • travis-ci joined the channel
      • 2019-03-01 06026, 2019

      • travis-ci
        Project bookbrainz-data-js build #984: failed in 3 min 37 sec: https://travis-ci.org/bookbrainz/bookbrainz-data-…
      • 2019-03-01 06026, 2019

      • travis-ci has left the channel
      • 2019-03-01 06004, 2019

      • Leo_Verto_ joined the channel
      • 2019-03-01 06016, 2019

      • Leo_Verto has quit
      • 2019-03-01 06016, 2019

      • Leo_Verto_ is now known as Leo_Verto
      • 2019-03-01 06053, 2019

      • Rotab has quit
      • 2019-03-01 06040, 2019

      • pulkit6559 joined the channel
      • 2019-03-01 06030, 2019

      • Slurpee has quit
      • 2019-03-01 06032, 2019

      • Rotab joined the channel
      • 2019-03-01 06007, 2019

      • Nyanko-sensei joined the channel
      • 2019-03-01 06059, 2019

      • D4RK-PH0ENiX has quit
      • 2019-03-01 06017, 2019

      • Freso
        Well, there's no link tax yet, so it's still free to link to things. :)
      • 2019-03-01 06022, 2019

      • silverstar joined the channel
      • 2019-03-01 06037, 2019

      • Nyanko-sensei has quit
      • 2019-03-01 06041, 2019

      • D4RK-PH0ENiX joined the channel
      • 2019-03-01 06029, 2019

      • michelv joined the channel
      • 2019-03-01 06012, 2019

      • reosarevok
        For at least one more month
      • 2019-03-01 06051, 2019

      • pristine_
        Moin moin!!
      • 2019-03-01 06029, 2019

      • pulkit6559 has quit
      • 2019-03-01 06034, 2019

      • ruaok
        moooin!
      • 2019-03-01 06033, 2019

      • ruaok
        I have a very effective alarm clock: https://www.instagram.com/p/BudahRDIOkJ/
      • 2019-03-01 06004, 2019

      • ruaok
        (best played at full volume to get the proper effect, even 6 stories up)
      • 2019-03-01 06032, 2019

      • pulkit6559 joined the channel
      • 2019-03-01 06008, 2019

      • pristine_
        ruaok: ping
      • 2019-03-01 06015, 2019

      • ruaok
        boing!
      • 2019-03-01 06051, 2019

      • pristine_
        Lol
      • 2019-03-01 06015, 2019

      • pristine_
        Should we set any targets for Monday?
      • 2019-03-01 06025, 2019

      • pristine_
        I will work this Sat and sunday
      • 2019-03-01 06038, 2019

      • pristine_
        Since was afk on Thursday and Friday
      • 2019-03-01 06051, 2019

      • pristine_
        iliekcomputers: ^^
      • 2019-03-01 06021, 2019

      • ruaok
        just a reminder, we're not slave drivers. if you want to take the weekend off, you owe us nothing.
      • 2019-03-01 06027, 2019

      • ruaok
        make sure to enjoy your life, please.
      • 2019-03-01 06035, 2019

      • michelv has quit
      • 2019-03-01 06026, 2019

      • iliekcomputers
      • 2019-03-01 06026, 2019

      • pristine_
        ruaok: I want to, please :)
      • 2019-03-01 06026, 2019

      • pristine_
        Lol
      • 2019-03-01 06037, 2019

      • iliekcomputers
        Yes boss!
      • 2019-03-01 06056, 2019

      • ruaok
        such a great picture.
      • 2019-03-01 06003, 2019

      • pristine_
        iliekcomputers: woh!
      • 2019-03-01 06006, 2019

      • pristine_
        Picture is amazing!
      • 2019-03-01 06043, 2019

      • ruaok
        ok, let me get another coffee and start thinking about LB and whatnot.
      • 2019-03-01 06054, 2019

      • pristine_
        ruaok: sure:)
      • 2019-03-01 06003, 2019

      • pristine_
        Is coffee effective?
      • 2019-03-01 06036, 2019

      • ruaok
        normally is. especially after being woken from a deep sleep by a samba band.
      • 2019-03-01 06051, 2019

      • ruaok
        but those are the dangers of living in this city.
      • 2019-03-01 06056, 2019

      • ruaok
        ok, I see that we have a lot of piece in place now, ready for review.
      • 2019-03-01 06025, 2019

      • ruaok
        I'll work on getting people paid, doing reviews and then some follow page work with Mr_Monkey today.
      • 2019-03-01 06046, 2019

      • ruaok
        thinking ahead, we need to improve how we show graphs on lb.org
      • 2019-03-01 06020, 2019

      • ruaok
        right now we have a demo bar graph up -- in Delhi we talked about getting prettier graphs and hoping to enlist the help of chhavi_.
      • 2019-03-01 06032, 2019

      • Nyanko-sensei joined the channel
      • 2019-03-01 06054, 2019

      • ruaok
        on the more hard-core geeky side of things, continuing exploring yarn would be good.
      • 2019-03-01 06034, 2019

      • Nyanko-sensei has quit
      • 2019-03-01 06034, 2019

      • ruaok
        would yarn make it easier to schedule all of these resources, so we don't have to do this super difficult arithmetic? 😆
      • 2019-03-01 06028, 2019

      • samj1912
        Btw we use Oozie with yarn for workflow management and scheduling
      • 2019-03-01 06040, 2019

      • D4RK-PH0ENiX has quit
      • 2019-03-01 06052, 2019

      • pristine_
        ruaok: even if we use yarn....we need this arithmetic
      • 2019-03-01 06004, 2019

      • samj1912
      • 2019-03-01 06012, 2019

      • ruaok
        hi samj1912
      • 2019-03-01 06017, 2019

      • samj1912
        Hello
      • 2019-03-01 06019, 2019

      • ruaok
        that sounds dirty.
      • 2019-03-01 06021, 2019

      • pristine_
        Yarn is an application/cluster manager
      • 2019-03-01 06040, 2019

      • ruaok
        do we need it yet? or will soon?
      • 2019-03-01 06043, 2019

      • pristine_
        But we need to specify the eexecutor count and other things.
      • 2019-03-01 06003, 2019

      • samj1912
        ruaok: makes life much easier with Hadoop
      • 2019-03-01 06004, 2019

      • pristine_
        And now we have everything sought out.
      • 2019-03-01 06034, 2019

      • pristine_
        ruaok: not exactly sure. Yarn is mostly pre installed on hadoop
      • 2019-03-01 06045, 2019

      • ruaok
        we don't use hadoop much, except for storage.
      • 2019-03-01 06054, 2019

      • samj1912
        You use spark right?
      • 2019-03-01 06056, 2019

      • pristine_
        It is better application manager than spark standalone
      • 2019-03-01 06057, 2019

      • ruaok
        yes
      • 2019-03-01 06007, 2019

      • samj1912
        Hmm, well your call
      • 2019-03-01 06013, 2019

      • pristine_
        ruaok: we can integrate spark with yarn
      • 2019-03-01 06019, 2019

      • samj1912
        Gotta go
      • 2019-03-01 06026, 2019

      • samj1912
        o/
      • 2019-03-01 06030, 2019

      • ruaok
        bai
      • 2019-03-01 06044, 2019

      • ruaok
        well, if it is a good near-term goal, then work on that over the weekend?
      • 2019-03-01 06024, 2019

      • ruaok
        the next task after that would be to start working on the queries that massages the data so it can be consumed in the collaborative filtering algorithm.
      • 2019-03-01 06030, 2019

      • pristine_
        It provides flexibility. But whatever work we have at this moment we can do it with standalone too. We may need yarn in future ( I am not sure)
      • 2019-03-01 06054, 2019

      • pristine_
        Okay. Yarn then
      • 2019-03-01 06026, 2019

      • pristine_
        ruaok: I don't know about the algorithm
      • 2019-03-01 06035, 2019

      • pristine_
        Would you like to brief :)
      • 2019-03-01 06015, 2019

      • ruaok
      • 2019-03-01 06000, 2019

      • ruaok
        this script loads the data, massages it and then persists some data frames.
      • 2019-03-01 06017, 2019

      • ruaok
        it uses the old method, before we started using HDFS and started really getting a clue.
      • 2019-03-01 06029, 2019

      • ruaok
        all of this needs to be ported to our new setup.
      • 2019-03-01 06042, 2019

      • pristine_
        Oh okay.
      • 2019-03-01 06054, 2019

      • ruaok
        to create new intermediate tables that we can re-generate at any time.
      • 2019-03-01 06057, 2019

      • pristine_
        Should we use Rabbitmq for that as well?
      • 2019-03-01 06018, 2019

      • ruaok
        for now, let's keep it simple and learn.
      • 2019-03-01 06034, 2019

      • ruaok
        just a batch system to take the data as it is right now and generate intermediate tables.
      • 2019-03-01 06037, 2019

      • ruaok
        then next is this:
      • 2019-03-01 06037, 2019

      • pristine_
        If I get it right....we need to import data from server in PG?
      • 2019-03-01 06037, 2019

      • ruaok
      • 2019-03-01 06001, 2019

      • ruaok
        no, we've already done that. we have listens in spark.
      • 2019-03-01 06025, 2019

      • ruaok
        next are the steps to process that data to be ready to train models.
      • 2019-03-01 06043, 2019

      • ruaok
        the training of models is where we see train the machine learning algorithm.
      • 2019-03-01 06057, 2019

      • pristine_
        Umm...okay
      • 2019-03-01 06058, 2019

      • ruaok
        the final step is to use the model we've created and to make new playlists.
      • 2019-03-01 06013, 2019

      • ruaok
        e.g. make actual recommendations.
      • 2019-03-01 06047, 2019

      • pristine_
        I am trying to understand :)
      • 2019-03-01 06024, 2019

      • ruaok
        keep asking questions. :)
      • 2019-03-01 06049, 2019

      • pristine_
        Could you write in points and share in chronological order :)
      • 2019-03-01 06001, 2019

      • pristine_
        Like set up yarn....fetch queries.. ..etc etc
      • 2019-03-01 06037, 2019

      • ruaok puts on morpheus sunglasses and opens his hand
      • 2019-03-01 06046, 2019

      • samj1912
        Oh PS - ruaok in case you want to play with pyspark - use the jupyter pyspark driver for prototyping models/data analysis
      • 2019-03-01 06059, 2019

      • CatQuest is so divided about being woken up by samba :/
      • 2019-03-01 06005, 2019

      • ruaok
        you have two choices, pristine_.
      • 2019-03-01 06022, 2019

      • ruaok
        if you take the blue pill you wake up a free person and will remember nothing.
      • 2019-03-01 06033, 2019

      • CatQuest
        on one hand: awesome
      • 2019-03-01 06033, 2019

      • CatQuest
        on the other: fecking NOISE ლ(ಠ益ಠლ)
      • 2019-03-01 06040, 2019

      • ruaok
        if you take the red pill you see all the truth and will get added to our trello board.
      • 2019-03-01 06042, 2019

      • pristine_
        What? Lol
      • 2019-03-01 06052, 2019

      • ruaok
        you've seen the matrix, yes?
      • 2019-03-01 06053, 2019

      • CatQuest
        pristine_: matrix meme
      • 2019-03-01 06001, 2019

      • pristine_
        No.
      • 2019-03-01 06007, 2019

      • pristine_
        Shit
      • 2019-03-01 06008, 2019

      • CatQuest
        ruaok: matrix is already old af
      • 2019-03-01 06018, 2019

      • iliekcomputers
        You're old af
      • 2019-03-01 06019, 2019

      • CatQuest
        need soemthinmore current yada
      • 2019-03-01 06027, 2019

      • ruaok
      • 2019-03-01 06036, 2019

      • CatQuest is 37 :V
      • 2019-03-01 06002, 2019

      • CatQuest
        that's old i gues. but not af o___O
      • 2019-03-01 06011, 2019

      • iliekcomputers
        Kidding :)
      • 2019-03-01 06012, 2019

      • pristine_
        Can someone tell me what does it mean?
      • 2019-03-01 06024, 2019

      • ruaok
        watch the video.
      • 2019-03-01 06025, 2019

      • CatQuest
        it means like one ting yo ucna go away free
      • 2019-03-01 06027, 2019

      • silverstar has quit
      • 2019-03-01 06032, 2019

      • CatQuest
        theo ther you will sink in deep
      • 2019-03-01 06034, 2019

      • pristine_
        ruaok: I will return in a while.
      • 2019-03-01 06054, 2019

      • pristine_
        It will be great if you make the points.
      • 2019-03-01 06006, 2019

      • ruaok
        tell me which pill you choose.
      • 2019-03-01 06014, 2019

      • pristine_
        Okay. Red
      • 2019-03-01 06016, 2019

      • iliekcomputers
        Green
      • 2019-03-01 06019, 2019

      • CatQuest
        hah
      • 2019-03-01 06024, 2019

      • CatQuest
        orange!
      • 2019-03-01 06035, 2019

      • ruaok
      • 2019-03-01 06036, 2019

      • iliekcomputers
        Ooh
      • 2019-03-01 06059, 2019

      • pristine_
        Blue
      • 2019-03-01 06001, 2019

      • Gazooo has quit
      • 2019-03-01 06014, 2019

      • ruaok
        once you chose red, you cannot go back.