#metabrainz

Project bookbrainz-data-js build #984: failed in 3 min 37 sec: https://travis-ci.org/bookbrainz/bookbrainz-dat...

Well, there's no link tax yet, so it's still free to link to things. :)

For at least one more month

Moin moin!!

moooin!

I have a very effective alarm clock: https://www.instagram.com/p/BudahRDIOkJ/

(best played at full volume to get the proper effect, even 6 stories up)

ruaok: ping

boing!

Lol

Should we set any targets for Monday?

I will work this Sat and sunday

Since was afk on Thursday and Friday

iliekcomputers: ^^

just a reminder, we're not slave drivers. if you want to take the weekend off, you owe us nothing.

make sure to enjoy your life, please.

ruaok: I want to, please :)

Lol

Yes boss!

such a great picture.

iliekcomputers: woh!

Picture is amazing!

ok, let me get another coffee and start thinking about LB and whatnot.

ruaok: sure:)

Is coffee effective?

normally is. especially after being woken from a deep sleep by a samba band.

but those are the dangers of living in this city.

ok, I see that we have a lot of piece in place now, ready for review.

I'll work on getting people paid, doing reviews and then some follow page work with Mr_Monkey today.

thinking ahead, we need to improve how we show graphs on lb.org

right now we have a demo bar graph up -- in Delhi we talked about getting prettier graphs and hoping to enlist the help of chhavi_.

on the more hard-core geeky side of things, continuing exploring yarn would be good.

would yarn make it easier to schedule all of these resources, so we don't have to do this super difficult arithmetic? 😆

Btw we use Oozie with yarn for workflow management and scheduling

ruaok: even if we use yarn....we need this arithmetic

hi samj1912

Hello

that sounds dirty.

Yarn is an application/cluster manager

do we need it yet? or will soon?

But we need to specify the eexecutor count and other things.

ruaok: makes life much easier with Hadoop

And now we have everything sought out.

ruaok: not exactly sure. Yarn is mostly pre installed on hadoop

we don't use hadoop much, except for storage.

You use spark right?

It is better application manager than spark standalone

yes

Hmm, well your call

ruaok: we can integrate spark with yarn

Gotta go

o/

bai

well, if it is a good near-term goal, then work on that over the weekend?

the next task after that would be to start working on the queries that massages the data so it can be consumed in the collaborative filtering algorithm.

It provides flexibility. But whatever work we have at this moment we can do it with standalone too. We may need yarn in future ( I am not sure)

Okay. Yarn then

ruaok: I don't know about the algorithm

Would you like to brief :)

this script loads the data, massages it and then persists some data frames.

it uses the old method, before we started using HDFS and started really getting a clue.

all of this needs to be ported to our new setup.

Oh okay.

to create new intermediate tables that we can re-generate at any time.

Should we use Rabbitmq for that as well?

for now, let's keep it simple and learn.

just a batch system to take the data as it is right now and generate intermediate tables.

then next is this:

If I get it right....we need to import data from server in PG?

no, we've already done that. we have listens in spark.

next are the steps to process that data to be ready to train models.

the training of models is where we see train the machine learning algorithm.

Umm...okay

the final step is to use the model we've created and to make new playlists.

e.g. make actual recommendations.

I am trying to understand :)

keep asking questions. :)

Could you write in points and share in chronological order :)

Like set up yarn....fetch queries.. ..etc etc

Oh PS - ruaok in case you want to play with pyspark - use the jupyter pyspark driver for prototyping models/data analysis

you have two choices, pristine_.

if you take the blue pill you wake up a free person and will remember nothing.

on one hand: awesome

on the other: fecking NOISE ლ(ಠ益ಠლ)

if you take the red pill you see all the truth and will get added to our trello board.

What? Lol

you've seen the matrix, yes?

pristine_: matrix meme

No.

Shit

ruaok: matrix is already old af

You're old af

need soemthinmore current yada

that's old i gues. but not af o___O

Kidding :)

watch the video.

it means like one ting yo ucna go away free

theo ther you will sink in deep

ruaok: I will return in a while.

It will be great if you make the points.

tell me which pill you choose.

Okay. Red

Green

hah

orange!

Ooh

Blue

once you chose red, you cannot go back.