#metabrainz

Epic won't disappear from other projects though, it is a known thing.

sure!

I was talking solely about inst

I am *curious* about how the epic ticket type functions

that was it

So, instead of "Epic" or "Mini-version", you do want "Research" (or some other name?) with labels field?

... no

i want

I'd like "Mini-version" but perhaps with a labels field for it?

liek the "epic" ting has

iliekcomputers: I was thinking the favourite artists and releases can also be shown graphically like a circular graph. What do you think about it

Epic is what iliekcomputers told you: an epic goal that usual depends on several other tickets.

yea yea yea sure but it needs ot be worked with to be understood

But what “Mini-version” means?

I mean. "Mini-version" is something I made reo make for me. idk why it ended up no other projects too

shivam-kapila: circular graph?

it was just a "min-version" because (at the time) I didn't know what epic did. nore did i have any editing for it

It ended up no other project because nobody understand what it is, there is no description attached to it.

¯\_(ツ)_/¯

i'm sorry abotu that

No problem, so mini-version = epic?

basically "Mini-version" is a ticket that collects loads of related tickets

that's waht I was wodnerin if maybe was a good idea :D when i heard tis epic description

!

so if "Mini-version" could have the *abilities* that epic has.. lables

maybe

anyway i cant really do anything epic on isnt right now. i don't have that label field

inst*

see https://tickets.metabrainz.org/browse/INST-392

INST-392: Veena

I mean if https://tickets.metabrainz.org/browse/INST-392 is a problematic name

ehr

if "Mini-version" is a problematic name..

iliekcomputers: something like this. I am not sure can it be called a pie chart. https://usercontent.irccloud-cdn.com/file/SAqvB...

[listenbrainz-server] dependabot-preview[bot] opened pull request #711 (master…dependabot/pip/py4j-0.10.9): Bump py4j from 0.10.8.1 to 0.10.9 https://github.com/metabrainz/listenbrainz-serv...

Hey pristine__. For this issue https://tickets.metabrainz.org/projects/LB/issu..., what do we exactly want to change in this file https://github.com/metabrainz/listenbrainz-serv...

LB-473: ListenBrainz-Labs: Use PySpark SQL Module in place of SQL queries.

Let us say we have a dataframe = df

We want to select col = user_id from this

There are two ways for that

Register the dataframe and then write sql query

`select user_id from df`

We are following this scheme in utils.py

Mr_Monkey: why exactly does all entities have 2 "language" selectors?

Now the other way out is don't register the dataframe but use pyspark sql module

Like this : `df.select("user_id")`

We want to update all the sql queries in utils.py to use pyspark module.

You will have a read the pyspark sql doc a bit. I have provided the link in the task.

Have we used this pyspark.sql module in any file in the listenbrainz_spark?

listenbrainz_spark/recommendations/create_dataframe.py

But I don't think referring to the file is the best way. You should read the doc once, it is really good :)

Yes sure. I will use the doc.

One more doubt!

Since we are passing a table to the function, do I need to now pass dataframe as a parameter?

or maybe just use this `sqlContext.table("table")` to convert it to a dataframe ?

Also pristine__, I dont see any queries listenbrainz_spark/recommendations/create_dataframe.py in this file. '=D

This script is calling other modules where queries are actually written

You have a lot of time. Please read the code.

Wait. I will give you the line no. A sec

> or maybe just use this `sqlContext.table("table")` to convert it to a dataframe ?

I have no idea what this line of code is?

> Since we are passing a table to the function, do I need to now pass dataframe as a parameter?

Which function? Take your time. I will review when you make a patch :)

This piece will register the table as a df so that I can use df operations

Okay sure

No

That's the whole point. We don't want to register tables, that is why we are directly using dataframe

Did you follow the above examples?

Tables are not registered as dataframe, we register a dataframe as a view/table to query it.

yes exactly

What I am saying is, to perform df operations, I need to paas df as a parameter here, right?

Obviously!

Yup cool 😅

I will get started then

Purpose of utils.py is to hold all the queries together at one place.

Yeah. Nice

pristine__: iliekcomputers: For testing the stats module how much data should I download from the ftp server dumps?

Hey

So we are in that process but I can suggest you a workaround

Checkout to `all-changes-mapping` branch

replace this path to `http://ftp.musicbrainz.org/pub/musicbrainz/listenbrainz/incremental/`

replace this to `listenbrainz-listens-dump-99-20191107-000002-spark-incremental.tar.xz`

CatQuest: From what I can see on Veena INST-392, you should be able to transition, link, and label it. Am I missing something?

INST-392: Veena https://tickets.metabrainz.org/browse/INST-392

then invoke https://github.com/metabrainz/listenbrainz-serv...

and run `/usr/local/spark/bin/spark-submit spark_manage.py upload_listens`

yvanzo: sure..

but I can't use a "epic link"

The configs are for full dumps which are not ideal for local env. We are in process to make small dumps for local env which will be done soon. Till then you can use incremental dumps to for stats

pristine: Got it. Thanks :D

> and run `/usr/local/spark/bin/spark-submit spark_manage.py upload_listens

this should be run inside listenbrainz_playground_1 container.

shivam-kapila:

Okay.

`docker exec -it listenbrainzspark_playground_1 bash`

This is nice. No download required. Wow :p

We are trying to make it better. In time people will have to write just one command to fill in data on their local machine, and in case you want to test something on big machines, a single command to fill in big data.

thank you :)

CatQuest: thanks, got it :)

iliekcomputers: worked on the feature. it is working well on the linux build

pristine__: can we document the steps you listed somewhere (for now)

Readme?

shivam-kapila: worked for you?

pristine__: we have a HACKING.md file that seems perfect for stuff like this

Cool. Thanks.

Project bookbrainz-data-js build #1366: passed in 1 min 49 sec: https://travis-ci.org/bookbrainz/bookbrainz-dat...

Dumps are getting created again, BTW.

CatQuest: should work now, but the migration is going to be cumbersome.

yvanzo: can you take a look at https://github.com/metabrainz/musicbrainz-serve... when you have some time?