#metabrainz

Morning!

Today's a good day to have a good day.

mo''in'

woohoo, file reading is okay. :)

>File loaded in 0.93 seconds

>File processed in 68.25 seconds

>Listens / sec = 27870.46

moooin!

27k listens a sec, hmmm, ok.

did the import ever finish?

ruaok: moin!

it wrote a bunch of listens and died around 2018/june for some reason.

running another one with more time and logging.

but it started with 200x?

yeah, it wrote dataframes for until 2017 (starting from 2002)

i checked them and they were valid.

promising!

hey guys, how old is picard?

like we should celebrate 10 yr anniversary soon maybe

yvanzo: I have made substantial changes to the draft. I have included a more detailed workflow and elaborated my deadline. I have to make more changes but before it. Please review and suggest if I am headed on the right track. Here's the link: https://community.metabrainz.org/t/gsoc-2019-br...

CatQuest: more than 10 years.

picard was started when I was living in a place where I lived 2003 - 2007.

ruaok: ok to work on spark stuff today?

oh no

anyway we should look at the ancientest commit messages and see whne it was. thne do some sort of celebration for 15 years of picard or thel ike

(depending on whne it was it might have to wait until 20 years)

but i want to do something

it's so cool

i remember whne picard was still a pipedream :D

any excuse to party right?

gelato or cake and a banner on the blog :D

chavi could design it

or m idk

sorry chhavi

iliekcomputers: yes, ready.

sorry for the late start.

No worries at all. :)

on my radar: line by line code review, volume/job support

Yes, cool.

where should we start?

do you have some output from the most recent script to share?

I added times, let me pull up some stats on average.

raw dump for me please.

or raw output.

"don't whitewash it jim, I can take it!"

iliekcomputers: surely I will.

good luck on your gsoc proposal amCap1712

It contains a lot of spark info.

CatQuest: thanks

One sec

so umm, it started writing the dataframes and the backlog is gone because of spark logs...

ok, whatever you got.

anyways, i was monitoring it continuously and it was taking around 70s on average for each listen file.

this includes the upload to hdfs etc too

the https://github.com/metabrainz/listenbrainz-reco...

getting to (https://github.com/metabrainz/listenbrainz-reco...) doesn't take much time 0.1s around on average.

but that is probably because spark lazy loads stuff.

listens/sec here (https://github.com/metabrainz/listenbrainz-reco...) were aroudn 40k

spark is very chatty, i think we might benefit from disabling INFO logs and running once.

the first two links are the same.

was that intentional?

hmm, sorry i pasted it without the message initially.

i added timings for how much time it takes to write a dataframe but all that is lost

in a sea of this https://www.irccloud.com/pastebin/pKb5J5ig/

ok, let's do that. we need to have more sane logs.

do you have a job running right now?

yeah.

have you estimated how long it will take to complete?

it's done all the processing and is just writing the dataframes now

done till 2012

how many hours so far?

should we thread this script?

2.9 h

then we can pick the number of threads to run concurrently.

or we have one thread do the processing and one thread to the writing data frames.

lets tune down the log messages.

then lets do a run over a few files to gather stats. then we can follow the code and see how long things take.

then we can determine how to improve it.

ok, I'll stop the code now.

make changes to run over ~20 files (the entire dump has ~100)

?

make it 10.

yokay. cool.

iliekcomputers: did you do something on lemmy that causes a lot of disk usage?

haven't touched lemmy in a while.

there was a massive spike in disk usage. got to 96%.

zas? you?

when?

new dumps were created today...

oh duh.

today is the 15th.

yeah.

well, still scary.

I'm cleaning old containers.

can you please log in and see if you have random dumps laying around in your home dir you no longer need?

alright.

i did have dumps in there, cleaned up.

much better. 21% reduction.

thanks!

no worries, sorry.

i've pushed the test code. let me see how to turn spark logs to ERROR.

how do you normally invoke Dockerfile.jobs?

ruaok: i push it and then have a script do a `docker run`.