#metabrainz

ruaok, idea for your hack weekend: https://newsroom.spotify.com/2020-09-29/how-to-...

pristine___: ping

_lucifer: pong

pristine___: i am getting java oom errors. what should i set driver memory for spark as?

There is no ideal value as such. Depends on your machine. You will have to tweak configs to get the right value for your machine. https://stackoverflow.com/questions/53631853/sp....

This might help.

You will have to calculate driver memory, excutor memory and other configs based on your machine. Fun maths :p

lol ok :)

what are these value for your machine btw?

I use few MBs of data so doesn't matter :p

yeah right :|

You can ishaanshah , I guess he also using full dumps.

ishaanshah: ping :)

pristine___: btw, have tried out google colab?

Not yet

But say? What's in your mind?

i was thinking if we could set up a jupyter notebook for quickly experimenting with reca

recs.

using colab, we may be able to run workloads using k80 gpus so speed and memory will less of an issie

_lucifer: pong!

ishaanshah: hi, do you use full dumps or incremental dumps locally while working eith spark?

multiple incremental dumps, not full

ah ok

i am too using that for listens

but for the mapping a full dump

yeah I used full for mapping too

but got OOM

yeah same here

were you able to tweak the config to get it working?

I have a 8G laptop

and the mapping is 11G

i too have a 8 gig one

so I dont think theres any way we can fix it, using a smaller dump would be better

yeah right that would certainly fix this

_lucifer: can do, next week when I am back in town

great!

_lucifer: told ya to not use full dump mapping :p

yeah you were right :)

Though I think we really need to have smaller dumps for dev

i have run it over and am monitoring to see where it fails

For better user experience.

_lucifer: How much memory do we get for free on colab?

Can you open a ticket for smaller dumps _lucifer ?

ishaanshah: i was trying to find the same

pristine___: yeah sure will do that

I am interested in having some kind of cloud testing env for spark

12gig

I used databricks, but it has limitations for free accounts

and doesnt work for full dumps

yeah right

> 12gig

:(

how much do you get on databrixks?

15G

but limited storage

so cant download the dumps

okay but that 12 does not include gpu

i'll see if i can find another alternative

ishaanshah: what about kaggle, 16 g + 30h gpu/week

_lucifer: Hmm, looks promising, I haven't personally used kaggle though

does it support spark?

Gcp gives upto 26g on demand

ishaanshah: yup, pyspark is just like any other mllib. i just installed pyspark on pc and experimented with using python console

lol, gcp banned my account

Good

_lucifer: ooh nice, let me know if you are able to run mapping on kaggle

yeah, will try and let you know :D

I like to experiment with the queries before writing it for production, notebooks like environment are good for this

ishaanshah: you mentioned about zepplin once

Wont it solve the issue if we have a zepplin layer in prod

shivam-kapila: yes but it requires you to use your own PC

Ouch

I dont have a powerful enough pc to join huge datasets

we can add it to prod but its not an easy task

Mine is slower than yours

thats true for almost all of us

Yeah I saw zepplin integration

Its somewhat tedious

Anyways I think a smaller mapping is needed

also a listen dataset for that

Cloud isnt as flex

so that the listens are actually in the mapping and we can get meaningful.resulta

yes that

ideally we can pick the latest 5 inc dumps and have corresponiding mapping

IG thats enough

yeah makes sense

_lucifer: this has some functions to download files from FTP and extracting them, maybe helpful https://usercontent.irccloud-cdn.com/file/mzgcs...

but getting that corresponding mapping can be hard

dunno think o

ishaanshah: thanks, i was just going to write these myself. a lot of time saved :D

:D

theres a dedicated spark extension for jupyter notebook

nice!

pristine___: ishaanshah took one hour but request dataframes completed succesfully so issue is not with the mapping

Dataframes created in an hour?

yeah

_lucifer: on kaggle or on local dev?

local

ok my bad it was 2 hours

but succesful

Oh the full mapping worked?

What changes did you make?

To the config

none

You said you got an OOM at first right?

i too had thought the issue was mapping but i had issued all command the last time

this time i am running all commands one by one as they complete

there are three left one of which should be the culprit

Oh

I dont know why it ran out of memory when I did it