So we need some directories in HDFS for storing models, dataframes and other things
Shouldn't we have a manage.py for them like we have it for dbs in LB server?
So that before we start with anything we have required directories in there
But this is for local setup, I am not sure how it works when running on server
ruaok
> But this is for local setup, I am not sure how it works when running on server
can you elaborate what you mean by this?
pristine__
I mean we won't be running manage.py on server right? It would drop the existing directories and create new one, and we will lost our data.
How do we ensure that all required directories are already created and we dont have to explicitly do mkdir from the scripts like train_model, create_dataframe etc
Lose*
ruaok
which server?
I would create a manage.py script, as you suggested, that we run on leader.
and if the directories already exist, stop doing anything.
pristine__
So we need to add it's path in start_master.sh?
So whenever we restart it executes?
ruaok
no, it should be a separate scrip that we invoke when we setup a new cluster...
speaking on which, I think I am going to order 8 dedicated machines from hetzner for a dedicated cluster.
that is always on.
akhilesh
Mr_Monkey: hi!
ruaok
and I think that in order to bootstrap that, all we need to do is add the new nodes to the swarm, let them replicate and then get rid of the old nodes.
pristine__
Swarm?
> Order 8 dedicated machines from hetzner
?
ruaok
our current cluster of 4 cloud instances are not particularly good. renting dedicated machines will make our data processing stuff a lot faster.
and since our machines are in a docker swarm, we can extend the cluster, let everything sync, then told off the old cloud instances.
pristine__
Okay. I am getting it. I will read cluster.py and I am sure I will have some doubts, I will ping you
reosarevok
iliekcomputers, ruaok: please, can we up the priority of decoupling LB usernames and profiles? I have another case of someone whose username changed in MB but I guess we can't change in LB. Unless you have a mechanism even if it is by hand-editing the DB? :)
ruaok
we now have a mechanism. but it isn't automated.
open a ticket for the specific user. we have a ticket for decoupling the usernames? if so, make it a higher priority and assign to iliekcomputers
LB-383: Allow updating usernames when they're changed in MusicBrainz
CatQuest
why did I read tree shat orcs first ¬_¬
reosarevok
Maybe it did. Maybe it did.
Mr_Monkey
Hi akhilesh ! Had some questions?
akhilesh
Mr_Monkey: Resolved now, but your help will require you create entities for browse endpoint tests.
CatQuest
oh shi waa supposed to ytest that app. right
Mr_Monkey
Sure akhilesh, let me know what you need
CatQuest
amCap1712: i tested it!
as expected the interface is... lacking :D
however i came across o crashing type bugs
no*
D4RK-PH0ENiX has quit
the "fingerprint" option seems to work as exepcted! it shows some generally logical selection
akhilesh
Mr_Monkey: I will try to complete tests today for browse requests, then I will inform you.
Mr_Monkey
OK
CatQuest
a problem is "metadata" button. it shows only random selections? (in any case i have yet to find anythingu se full for it)
it's not clear that saving works, (and I see no copied file in a folder underneath)
(feature requests: be able to manually edit tags, be able to select tags to "keep old tag", selecting more than one file at a time(!), in reference to that; be able to select which release in a release group to tag with)
especially this for coverart
ok, keep up the good work for now!
D4RK-PH0ENiX joined the channel
amCap1712
CatQuest: hi
i know the problem with metadata but am unable to find the root cause
CatQuest: also the file is saved in the device public storage directory
D4RK-PH0ENiX has quit
D4RK-PH0ENiX joined the channel
iliekcomputers has quit
iliekcomputers joined the channel
BrainzGit
[listenbrainz-labs] vansika closed pull request #44 (master…develop-sh): Modify develop.sh to use a single command for building services and starting containers https://github.com/metabrainz/listenbrainz-labs...
ruaok
pristine__: are you using the cluster right now or can I test some stuff?
(that will require me starting/stopping the cluster.)
pristine__
I am not using the cluster.
Go ahead :)
ruaok
ok
travis-ci joined the channel
travis-ci
metabrainz/picard#4780 (master - 41c481e : Philipp Wolfer): The build passed.
ruaok: people who want to set up lb labs on their local machine won't be running a script for setting up a cluster?
They just do develop.sh build, right?
And install depedencies as stated in readme
ruaok
well, local machine != cluster
so, really it is up to us to decide what to do and how to do it. but it might be a while before someone follows our steps and setups up their own cluster
pristine__
Yes. So ideally manage.py for them should be invoked with develop.sh
And on leader with setup_cluster.py
no?
My next PR involves making a new directory so I want to get it right.
ruaok
setup_cluster will not be supported anymore.
once we setup our cluster of dedicated machines, we'll leave the cloud instances behind.
and create_cluser will not be useful.
I think manage.py should just be invoked separately from develop.sh
pristine__
Okay. We can include thay in readme to invoke it separately?
What do you mean by cloud instances here if I may ask?
pristine__ apologizes for asking so many questions
ruaok
right now we rent 4 servers from hetzner cloud. those isntances are not very good for our needs.
the classic hetzner service allows us to rent a dedicated server where we have full control.
no one can take CPU power away from us.
pristine__
Okay.
So you remember about the error I posted saying I am not able to resolve because it says check driver logs and RPC disassociated all the time?
ruaok
sort of yes?
the cluster is up again. can you run a quick test program that runs one simple spark query to see if things are working?