Hi2All. My question is a little bit offtopic, but i am just asking here, cause here are developers and may be u could give me some advice for the noob like me. I am trying to make a java script, which could parse GEMA PRO database and extract track names with details. PM me if u are interested please.
2019-07-25 20602, 2019
pristine__
ruaok: moin. Can we create a tunnel through worker nodes. Stderr and stdout for workers are visible on 8081
2019-07-25 20658, 2019
ruaok
moin!
2019-07-25 20610, 2019
ruaok
visible on port 8081 inside the container?
2019-07-25 20653, 2019
pristine__
We have created a tunnel for port 4040 running in listenbrainz jobs on leader. This is the Spark UI. Now in this UI there are tabs (stderr, stdout) for every executor to view executor logs. When I click on this, a window opens, and something like *10.x.x.5x:8081.xxxxx* is written in address bar but nothing is displayed.
2019-07-25 20612, 2019
pristine__
So I guess we need a tunnel for 8081.
2019-07-25 20625, 2019
ruaok
ok, yes that means a tunnel into a different machine and then into the container. hmm. I'll have to ponder this.
2019-07-25 20654, 2019
pristine__
Yeah. We just want executor logs to debug. Now, by default spark logs in spark_home/logs. On leader, spark/logs in empty. Idk why. I have spent days on this but idk.
2019-07-25 20619, 2019
pristine__
By default workers store stderr and stdout in spark_home/work
2019-07-25 20645, 2019
pristine__
Can you log in to one of the worker and check /usr/local/spark/work
2019-07-25 20647, 2019
pristine__
?
2019-07-25 20645, 2019
ruaok
that might be easier.
2019-07-25 20648, 2019
ruaok
any worker?
2019-07-25 20608, 2019
pristine__
I have created a requested to join spark users list, I can ask about empty logs there. UI serves the same purpose, but it will stop once the job stops.
2019-07-25 20612, 2019
pristine__
Yeah. Any worker.
2019-07-25 20655, 2019
pristine__
But I still say that we should try for workers UI. It is better to apprehend. The stored logs (which must be cleaned regularly) can help us too see error history even if the job has ended. Whislt the job is running, UI can be great.
2019-07-25 20626, 2019
ruaok
lets see if the files are useful first.
2019-07-25 20642, 2019
ruaok
on leader in /home/vansika is worker-logs.zip
2019-07-25 20619, 2019
pristine__
A sec
2019-07-25 20618, 2019
pristine__
ruaok: can you go to /usr/local/spark and send me a screenshot
2019-07-25 20642, 2019
pristine__
And then /usr/local/spark/work and a screenshot
2019-07-25 20610, 2019
ruaok
you have all of the contents of the work directory in the zip file. did the zip file not work?
Cool. What about app I ran yesterday? Are they stored on other workers? We don't have logs of previous week/month. Are they cleaned up automatically?
2019-07-25 20609, 2019
pristine__
So many questions. Lol
2019-07-25 20656, 2019
ruaok
I presume the other workers have similar files. and I would expect them to get cleaned up after x days, which is likely configurable in some config file.
2019-07-25 20602, 2019
ruaok
is the output useful?
2019-07-25 20652, 2019
pristine__
Loads of it. I will go through it and get back to you
2019-07-25 20605, 2019
pristine__
Can we discuss a lil about cleaning up models?
2019-07-25 20615, 2019
pristine__
(thanks for the zip)
2019-07-25 20630, 2019
ruaok
sure.
2019-07-25 20653, 2019
pristine__
So i was thinking, whenever we save a model (which will be after months most probably) should we clean up the previous one(s)?
2019-07-25 20628, 2019
ruaok
I wonder what our thinking here should be.
2019-07-25 20641, 2019
ruaok
clean up by default and mark others as "saved, in use"?
2019-07-25 20656, 2019
ruaok
or manually clean up and only carefully delete items.
2019-07-25 20634, 2019
ruaok
or maybe, just keep the latest one? I guess the question is how we specify which module to use for recommendations.
2019-07-25 20647, 2019
pristine__
The latest one
2019-07-25 20653, 2019
pristine__
I guess
2019-07-25 20619, 2019
ruaok
perhaps, this question is premature.
2019-07-25 20625, 2019
pristine__
If we dont delete, we may run out of space in time.
2019-07-25 20637, 2019
ruaok
it is a good question, but we're not fully certain how we're going to use data yet.
2019-07-25 20641, 2019
ruaok
we will.
2019-07-25 20604, 2019
ruaok
how about we do something simple to start with and simply keep the X latest models, but delete everything else?
2019-07-25 20619, 2019
pristine__
Cool. So maybe till we are sure about it, i will manual delete all the models which are created while testing.
2019-07-25 20631, 2019
pristine__
Yes. Sound good.
2019-07-25 20633, 2019
ruaok
ok
2019-07-25 20607, 2019
pristine__
What should X be?
2019-07-25 20644, 2019
ruaok
7?
2019-07-25 20606, 2019
pristine__
Ummm. It actually depends on how much data we are using for training. Like for around 6 months, consider one gb. 1*7 = 7gb
2019-07-25 20613, 2019
pristine__
7*3 = 21gb
2019-07-25 20618, 2019
pristine__
After replication.
2019-07-25 20614, 2019
pristine__
Also, we should have a json or parquet for storing matadata about models (on which data it eas trained, when trained size etc etc.)
2019-07-25 20632, 2019
pristine__
Maybe 4 to start with.
2019-07-25 20641, 2019
ruaok
:)
2019-07-25 20609, 2019
pristine__
I was just thinking, why would we require folder models? Could not get to an answer?
2019-07-25 20617, 2019
pristine__
Older*
2019-07-25 20648, 2019
ruaok
we may find a model that works well and put it into production.
2019-07-25 20602, 2019
ruaok
but at the same time we will want to continue evolving models.
2019-07-25 20619, 2019
pristine__
Oh. Right.
2019-07-25 20626, 2019
ruaok
we need to keep at least one around for production. possible keep more around for various production scenarious.
2019-07-25 20624, 2019
pristine__
This project is wow. Everything has to be done from scratch, so much brainstorming. Yay! Thank you <3
2019-07-25 20602, 2019
ruaok
I know the feeling. part of it is exciting, part of it is tiring. but it has been tiring for 20 years, so I am used to it.
2019-07-25 20618, 2019
pristine__
I have never done something like this before. I like it.
