#metabrainz

/

      • niceplace has quit
      • niceplace joined the channel
      • niceplace has quit
      • niceplace joined the channel
      • CatQuest has left the channel
      • CatQuest joined the channel
      • CatQuest has quit
      • cats has quit
      • aidanlw17 has quit
      • kuno has quit
      • cats joined the channel
      • aidanlw17 joined the channel
      • kuno joined the channel
      • CatQuest joined the channel
      • akhilesh
        Moin!
      • Gazooo has quit
      • Gazooo joined the channel
      • pristine__
        ruaok: hey. moin
      • ruaok
        moin moin!
      • I haz coffee. can chat.
      • pristine__
        Nice
      • So we need some directories in HDFS for storing models, dataframes and other things
      • Shouldn't we have a manage.py for them like we have it for dbs in LB server?
      • So that before we start with anything we have required directories in there
      • But this is for local setup, I am not sure how it works when running on server
      • ruaok
        > But this is for local setup, I am not sure how it works when running on server
      • can you elaborate what you mean by this?
      • pristine__
        I mean we won't be running manage.py on server right? It would drop the existing directories and create new one, and we will lost our data.
      • How do we ensure that all required directories are already created and we dont have to explicitly do mkdir from the scripts like train_model, create_dataframe etc
      • Lose*
      • ruaok
        which server?
      • I would create a manage.py script, as you suggested, that we run on leader.
      • and if the directories already exist, stop doing anything.
      • pristine__
        So we need to add it's path in start_master.sh?
      • So whenever we restart it executes?
      • ruaok
        no, it should be a separate scrip that we invoke when we setup a new cluster...
      • speaking on which, I think I am going to order 8 dedicated machines from hetzner for a dedicated cluster.
      • that is always on.
      • akhilesh
        Mr_Monkey: hi!
      • ruaok
        and I think that in order to bootstrap that, all we need to do is add the new nodes to the swarm, let them replicate and then get rid of the old nodes.
      • pristine__
        Swarm?
      • > Order 8 dedicated machines from hetzner
      • ?
      • ruaok
        our current cluster of 4 cloud instances are not particularly good. renting dedicated machines will make our data processing stuff a lot faster.
      • and since our machines are in a docker swarm, we can extend the cluster, let everything sync, then told off the old cloud instances.
      • pristine__
        Okay. I am getting it. I will read cluster.py and I am sure I will have some doubts, I will ping you
      • reosarevok
        iliekcomputers, ruaok: please, can we up the priority of decoupling LB usernames and profiles? I have another case of someone whose username changed in MB but I guess we can't change in LB. Unless you have a mechanism even if it is by hand-editing the DB? :)
      • ruaok
        we now have a mechanism. but it isn't automated.
      • open a ticket for the specific user. we have a ticket for decoupling the usernames? if so, make it a higher priority and assign to iliekcomputers
      • reosarevok
      • BrainzBot
        LB-467: Rename user budkin to treeshateorcs
      • reosarevok
        I see it doesn't automatically assign it to anyone, should I?
      • ruaok
        yes, plz
      • reosarevok
        yes meaning "yes, to iliekcomputers" as well? :)
      • https://tickets.metabrainz.org/browse/LB-383 is the decoupling one, will assign
      • BrainzBot
        LB-383: Allow updating usernames when they're changed in MusicBrainz
      • CatQuest
        why did I read tree shat orcs first ¬_¬
      • reosarevok
        Maybe it did. Maybe it did.
      • Mr_Monkey
        Hi akhilesh ! Had some questions?
      • akhilesh
        Mr_Monkey: Resolved now, but your help will require you create entities for browse endpoint tests.
      • CatQuest
        oh shi waa supposed to ytest that app. right
      • Mr_Monkey
        Sure akhilesh, let me know what you need
      • CatQuest
        amCap1712: i tested it!
      • as expected the interface is... lacking :D
      • however i came across o crashing type bugs
      • no*
      • D4RK-PH0ENiX has quit
      • the "fingerprint" option seems to work as exepcted! it shows some generally logical selection
      • akhilesh
        Mr_Monkey: I will try to complete tests today for browse requests, then I will inform you.
      • Mr_Monkey
        OK
      • CatQuest
        a problem is "metadata" button. it shows only random selections? (in any case i have yet to find anythingu se full for it)
      • it's not clear that saving works, (and I see no copied file in a folder underneath)
      • (feature requests: be able to manually edit tags, be able to select tags to "keep old tag", selecting more than one file at a time(!), in reference to that; be able to select which release in a release group to tag with)
      • especially this for coverart
      • ok, keep up the good work for now!
      • D4RK-PH0ENiX joined the channel
      • amCap1712
        CatQuest: hi
      • i know the problem with metadata but am unable to find the root cause
      • CatQuest: also the file is saved in the device public storage directory
      • D4RK-PH0ENiX has quit
      • D4RK-PH0ENiX joined the channel
      • iliekcomputers has quit
      • iliekcomputers joined the channel
      • BrainzGit
        [listenbrainz-labs] vansika closed pull request #44 (master…develop-sh): Modify develop.sh to use a single command for building services and starting containers https://github.com/metabrainz/listenbrainz-labs...
      • ruaok
        pristine__: are you using the cluster right now or can I test some stuff?
      • (that will require me starting/stopping the cluster.)
      • pristine__
        I am not using the cluster.
      • Go ahead :)
      • ruaok
        ok
      • travis-ci joined the channel
      • travis-ci
        metabrainz/picard#4780 (master - 41c481e : Philipp Wolfer): The build passed.
      • travis-ci has left the channel
      • spuniun has quit
      • pristine__
        ruaok: people who want to set up lb labs on their local machine won't be running a script for setting up a cluster?
      • They just do develop.sh build, right?
      • And install depedencies as stated in readme
      • ruaok
        well, local machine != cluster
      • so, really it is up to us to decide what to do and how to do it. but it might be a while before someone follows our steps and setups up their own cluster
      • pristine__
        Yes. So ideally manage.py for them should be invoked with develop.sh
      • And on leader with setup_cluster.py
      • no?
      • My next PR involves making a new directory so I want to get it right.
      • ruaok
        setup_cluster will not be supported anymore.
      • once we setup our cluster of dedicated machines, we'll leave the cloud instances behind.
      • and create_cluser will not be useful.
      • I think manage.py should just be invoked separately from develop.sh
      • pristine__
        Okay. We can include thay in readme to invoke it separately?
      • What do you mean by cloud instances here if I may ask?
      • pristine__ apologizes for asking so many questions
      • ruaok
        right now we rent 4 servers from hetzner cloud. those isntances are not very good for our needs.
      • the classic hetzner service allows us to rent a dedicated server where we have full control.
      • no one can take CPU power away from us.
      • pristine__
        Okay.
      • So you remember about the error I posted saying I am not able to resolve because it says check driver logs and RPC disassociated all the time?
      • ruaok
        sort of yes?
      • the cluster is up again. can you run a quick test program that runs one simple spark query to see if things are working?
      • pristine__
        A sec
      • ruaok: ^
      • ruaok
        erp, not good.
      • pristine__
        hmm
      • ruaok
        try again?
      • pristine__
        A sec
      • same error
      • ruaok: I was going out to buy something, do you need me around? I can go after some time
      • ruaok
        go, I'll keep playing.
      • its there a command line you can give me to run what you run?
      • pristine__
        I just ran create_dataframes.
      • ./spark-submit manage.py create_dataframes
      • ruaok
        thx
      • pristine__
        :)
      • I will tell you about the error and sol later at night. See ya.
      • yvanzo
        ruaok: sir test vm is unresponsive again, can you please restart it?
      • ruaok
        on it
      • spuniun joined the channel
      • yvanzo: 104.197.183.152
      • yvanzo
        ruaok: thank you!
      • ruaok
        pristine__: looks better now, but the script ran into a different error.
      • have a look when you return?
      • pristine__
        Okay
      • ruaok
        I hope it works. I think the configuration is much more sane now.
      • yvanzo
        ruaok: is it possible to upgrade sir test vm?
      • ruaok
        how much work would it be for you to start with a new VM?
      • I can give you one from azure..
      • but if that is a pain, then I will resize the current one.
      • either way, let me know desired specs
      • yvanzo
        ruaok: it will be easy to start with a new one, possibly with 16 threads?
      • (or 8 threads at least)