10:54 AM
yvanzo
Epic won't disappear from other projects though, it is a known thing.
2020-01-27 02706, 2020
10:54 AM
CatQuest
sure!
2020-01-27 02716, 2020
10:54 AM
CatQuest
I was talking solely about inst
2020-01-27 02734, 2020
10:54 AM
CatQuest
I am *curious* about how the epic ticket type functions
2020-01-27 02746, 2020
10:54 AM
CatQuest
that was it
2020-01-27 02748, 2020
10:54 AM
yvanzo
So, instead of "Epic" or "Mini-version", you do want "Research" (or some other name?) with labels field?
2020-01-27 02755, 2020
10:54 AM
CatQuest
... no
2020-01-27 02757, 2020
10:54 AM
CatQuest
i want
2020-01-27 02719, 2020
10:55 AM
CatQuest
I'd like "Mini-version" but perhaps with a labels field for it?
2020-01-27 02724, 2020
10:55 AM
CatQuest
liek the "epic" ting has
2020-01-27 02731, 2020
10:55 AM
shivam-kapila
iliekcomputers: I was thinking the favourite artists and releases can also be shown graphically like a circular graph. What do you think about it
2020-01-27 02702, 2020
10:56 AM
yvanzo
Epic is what iliekcomputers told you: an epic goal that usual depends on several other tickets.
2020-01-27 02722, 2020
10:56 AM
CatQuest
yea yea yea sure but it needs ot be worked with to be understood
2020-01-27 02738, 2020
10:56 AM
yvanzo
But what “Mini-version” means?
2020-01-27 02741, 2020
10:56 AM
CatQuest
I mean. "Mini-version" is something I made reo make for me. idk why it ended up no other projects too
2020-01-27 02759, 2020
10:56 AM
iliekcomputers
shivam-kapila: circular graph?
2020-01-27 02721, 2020
10:57 AM
CatQuest
it was just a "min-version" because (at the time) I didn't know what epic did. nore did i have any editing for it
2020-01-27 02722, 2020
10:57 AM
yvanzo
It ended up no other project because nobody understand what it is, there is no description attached to it.
2020-01-27 02730, 2020
10:57 AM
CatQuest
¯\_(ツ)_/¯
2020-01-27 02735, 2020
10:57 AM
CatQuest
i'm sorry abotu that
2020-01-27 02751, 2020
10:57 AM
yvanzo
No problem, so mini-version = epic?
2020-01-27 02700, 2020
10:58 AM
CatQuest
basically "Mini-version" is a ticket that collects loads of related tickets
2020-01-27 02717, 2020
10:58 AM
CatQuest
that's waht I was wodnerin if maybe was a good idea :D when i heard tis epic description
2020-01-27 02718, 2020
10:58 AM
CatQuest
!
2020-01-27 02705, 2020
10:59 AM
CatQuest
so if "Mini-version" could have the *abilities* that epic has.. lables
2020-01-27 02710, 2020
10:59 AM
CatQuest
maybe
2020-01-27 02756, 2020
10:59 AM
CatQuest
anyway i cant really do anything epic on isnt right now. i don't have that label field
2020-01-27 02701, 2020
11:00 AM
CatQuest
inst*
2020-01-27 02712, 2020
11:00 AM
CatQuest
2020-01-27 02713, 2020
11:00 AM
BrainzBot
INST-392: Veena
2020-01-27 02731, 2020
11:00 AM
CatQuest
2020-01-27 02735, 2020
11:00 AM
CatQuest
ehr
2020-01-27 02745, 2020
11:00 AM
CatQuest
if "Mini-version" is a problematic name..
2020-01-27 02712, 2020
11:01 AM
CatQuest needs breakfast first
2020-01-27 02718, 2020
11:02 AM
shivam-kapila
2020-01-27 02708, 2020
11:08 AM
prabal joined the channel
2020-01-27 02737, 2020
11:14 AM
sarthak_jain joined the channel
2020-01-27 02743, 2020
11:17 AM
Nyanko-sensei has quit
2020-01-27 02718, 2020
11:22 AM
BrainzGit
2020-01-27 02714, 2020
11:23 AM
MajorLurker has quit
2020-01-27 02700, 2020
11:28 AM
Nyanko-sensei joined the channel
2020-01-27 02702, 2020
11:35 AM
sarthak_jain
2020-01-27 02703, 2020
11:35 AM
BrainzBot
LB-473: ListenBrainz-Labs: Use PySpark SQL Module in place of SQL queries.
2020-01-27 02754, 2020
11:35 AM
MajorLurker joined the channel
2020-01-27 02706, 2020
11:39 AM
sarthak_jain
We wish to return a dictionary right, from each function?
2020-01-27 02755, 2020
11:39 AM
pristine__
Let us say we have a dataframe = df
2020-01-27 02714, 2020
11:40 AM
pristine__
We want to select col = user_id from this
2020-01-27 02727, 2020
11:40 AM
pristine__
There are two ways for that
2020-01-27 02749, 2020
11:40 AM
pristine__
Register the dataframe and then write sql query
2020-01-27 02705, 2020
11:41 AM
pristine__
`select user_id from df`
2020-01-27 02724, 2020
11:41 AM
pristine__
We are following this scheme in utils.py
2020-01-27 02751, 2020
11:41 AM
CatQuest
Mr_Monkey: why exactly does all entities have 2 "language" selectors?
2020-01-27 02751, 2020
11:41 AM
pristine__
Now the other way out is don't register the dataframe but use pyspark sql module
2020-01-27 02741, 2020
11:42 AM
pristine__
Like this : `df.select("user_id")`
2020-01-27 02706, 2020
11:43 AM
pristine__
We want to update all the sql queries in utils.py to use pyspark module.
2020-01-27 02740, 2020
11:43 AM
pristine__
You will have a read the pyspark sql doc a bit. I have provided the link in the task.
2020-01-27 02757, 2020
11:43 AM
sarthak_jain
Have we used this pyspark.sql module in any file in the listenbrainz_spark?
2020-01-27 02743, 2020
11:44 AM
pristine__
listenbrainz_spark/recommendations/create_dataframe.py
2020-01-27 02751, 2020
11:45 AM
pristine__
But I don't think referring to the file is the best way. You should read the doc once, it is really good :)
2020-01-27 02703, 2020
11:46 AM
sarthak_jain
Yes sure. I will use the doc.
2020-01-27 02725, 2020
11:46 AM
sarthak_jain
One more doubt!
2020-01-27 02751, 2020
11:46 AM
sarthak_jain
Since we are passing a table to the function, do I need to now pass dataframe as a parameter?
2020-01-27 02722, 2020
11:48 AM
sarthak_jain
or maybe just use this `sqlContext.table("table")` to convert it to a dataframe ?
2020-01-27 02747, 2020
11:48 AM
Guest84633 is now known as c1e0
2020-01-27 02710, 2020
11:51 AM
sarthak_jain
Also pristine__, I dont see any queries listenbrainz_spark/recommendations/create_dataframe.py in this file. '=D
2020-01-27 02708, 2020
11:53 AM
pristine__
This script is calling other modules where queries are actually written
2020-01-27 02718, 2020
11:53 AM
pristine__
You have a lot of time. Please read the code.
2020-01-27 02736, 2020
11:53 AM
pristine__
Wait. I will give you the line no. A sec
2020-01-27 02747, 2020
11:54 AM
pristine__
2020-01-27 02713, 2020
11:55 AM
pristine__
> or maybe just use this `sqlContext.table("table")` to convert it to a dataframe ?
2020-01-27 02729, 2020
11:55 AM
pristine__
I have no idea what this line of code is?
2020-01-27 02742, 2020
11:55 AM
pristine__
> Since we are passing a table to the function, do I need to now pass dataframe as a parameter?
2020-01-27 02745, 2020
11:56 AM
pristine__
Which function? Take your time. I will review when you make a patch :)
2020-01-27 02705, 2020
11:57 AM
sarthak_jain
This piece will register the table as a df so that I can use df operations
2020-01-27 02724, 2020
11:57 AM
sarthak_jain
Okay sure
2020-01-27 02751, 2020
11:59 AM
pristine__
No
2020-01-27 02718, 2020
12:00 PM
pristine__
That's the whole point. We don't want to register tables, that is why we are directly using dataframe
2020-01-27 02728, 2020
12:00 PM
pristine__
Did you follow the above examples?
2020-01-27 02708, 2020
12:01 PM
pristine__
Tables are not registered as dataframe, we register a dataframe as a view/table to query it.
2020-01-27 02738, 2020
12:01 PM
sarthak_jain has quit
2020-01-27 02741, 2020
12:01 PM
Coder44 has quit
2020-01-27 02715, 2020
12:03 PM
sarthak_jain joined the channel
2020-01-27 02724, 2020
12:03 PM
sarthak_jain
2020-01-27 02732, 2020
12:03 PM
sarthak_jain
yes exactly
2020-01-27 02720, 2020
12:04 PM
sarthak_jain
What I am saying is, to perform df operations, I need to paas df as a parameter here, right?
2020-01-27 02744, 2020
12:04 PM
sarthak_jain has quit
2020-01-27 02703, 2020
12:05 PM
pristine__
Obviously!
2020-01-27 02735, 2020
12:06 PM
sarthak_jain joined the channel
2020-01-27 02719, 2020
12:07 PM
sarthak_jain
Yup cool 😅
2020-01-27 02737, 2020
12:07 PM
sarthak_jain
I will get started then
2020-01-27 02744, 2020
12:07 PM
sarthak_jain has quit
2020-01-27 02755, 2020
12:07 PM
pristine__
Purpose of utils.py is to hold all the queries together at one place.
2020-01-27 02700, 2020
12:08 PM
pristine__
Yeah. Nice
2020-01-27 02724, 2020
12:14 PM
shivam-kapila
pristine__: iliekcomputers: For testing the stats module how much data should I download from the ftp server dumps?
2020-01-27 02748, 2020
12:14 PM
pristine__
Hey
2020-01-27 02723, 2020
12:15 PM
pristine__
So we are in that process but I can suggest you a workaround
2020-01-27 02707, 2020
12:16 PM
pristine__
Checkout to `all-changes-mapping` branch
2020-01-27 02725, 2020
12:17 PM
pristine__
2020-01-27 02702, 2020
12:18 PM
pristine__
replace this path to `http://ftp.musicbrainz.org/pub/musicbrainz/listenbrainz/incremental/`
2020-01-27 02743, 2020
12:18 PM
pristine__
2020-01-27 02710, 2020
12:19 PM
pristine__
replace this to `listenbrainz-listens-dump-99-20191107-000002-spark-incremental.tar.xz`
2020-01-27 02732, 2020
12:20 PM
yvanzo
CatQuest: From what I can see on Veena INST-392, you should be able to transition, link, and label it. Am I missing something?
2020-01-27 02733, 2020
12:20 PM
BrainzBot
2020-01-27 02709, 2020
12:21 PM
pristine__
2020-01-27 02715, 2020
12:22 PM
pristine__
and run `/usr/local/spark/bin/spark-submit spark_manage.py upload_listens`
2020-01-27 02732, 2020
12:22 PM
CatQuest
yvanzo: sure..
2020-01-27 02745, 2020
12:22 PM
CatQuest
but I can't use a "epic link"
2020-01-27 02719, 2020
12:23 PM
pristine__
The configs are for full dumps which are not ideal for local env. We are in process to make small dumps for local env which will be done soon. Till then you can use incremental dumps to for stats
2020-01-27 02720, 2020
12:25 PM
shivam-kapila
pristine: Got it. Thanks :D
2020-01-27 02703, 2020
12:26 PM
pristine__
> and run `/usr/local/spark/bin/spark-submit spark_manage.py upload_listens
2020-01-27 02734, 2020
12:26 PM
pristine__
this should be run inside listenbrainz_playground_1 container.
2020-01-27 02736, 2020
12:26 PM
pristine__
shivam-kapila:
2020-01-27 02734, 2020
12:27 PM
shivam-kapila
Okay.
2020-01-27 02711, 2020
12:28 PM
pristine__
`docker exec -it listenbrainzspark_playground_1 bash`
2020-01-27 02708, 2020
12:31 PM
shivam-kapila
This is nice. No download required. Wow :p
2020-01-27 02703, 2020
12:34 PM
pristine__
We are trying to make it better. In time people will have to write just one command to fill in data on their local machine, and in case you want to test something on big machines, a single command to fill in big data.
2020-01-27 02708, 2020
12:34 PM
pristine__
thank you :)
2020-01-27 02753, 2020
12:34 PM
antlarr has quit
2020-01-27 02724, 2020
12:35 PM
yvanzo
CatQuest: thanks, got it :)
2020-01-27 02711, 2020
12:36 PM
antlarr joined the channel
2020-01-27 02734, 2020
12:56 PM
amCap1712
iliekcomputers: worked on the feature. it is working well on the linux build
2020-01-27 02700, 2020
13:01 PM
c1e0_ joined the channel
2020-01-27 02737, 2020
13:03 PM
c1e0 has quit
2020-01-27 02733, 2020
13:07 PM
iliekcomputers
pristine__: can we document the steps you listed somewhere (for now)
2020-01-27 02700, 2020
13:08 PM
pristine__
Readme?
2020-01-27 02755, 2020
13:08 PM
pristine__
shivam-kapila: worked for you?
2020-01-27 02742, 2020
13:11 PM
iliekcomputers
pristine__: we have a HACKING.md file that seems perfect for stuff like this
2020-01-27 02757, 2020
13:11 PM
iliekcomputers
2020-01-27 02711, 2020
13:12 PM
pristine__
Cool. Thanks.
2020-01-27 02750, 2020
13:27 PM
prabal has quit
2020-01-27 02733, 2020
13:59 PM
travis-ci joined the channel
2020-01-27 02733, 2020
13:59 PM
travis-ci
2020-01-27 02733, 2020
13:59 PM
travis-ci has left the channel
2020-01-27 02743, 2020
14:19 PM
Lotheric has quit
2020-01-27 02712, 2020
14:21 PM
iliekcomputers
Dumps are getting created again, BTW.
2020-01-27 02730, 2020
14:24 PM
sarthak_jain joined the channel
2020-01-27 02706, 2020
14:30 PM
Lotheric joined the channel
2020-01-27 02729, 2020
14:48 PM
shivam-kapila has quit
2020-01-27 02757, 2020
15:06 PM
yvanzo
CatQuest: should work now, but the migration is going to be cumbersome.
2020-01-27 02754, 2020
15:30 PM
rahul24 joined the channel
2020-01-27 02748, 2020
15:36 PM
rahul24 has quit
2020-01-27 02756, 2020
15:38 PM
rahul24 joined the channel
2020-01-27 02725, 2020
15:45 PM
rahul24 has quit
2020-01-27 02703, 2020
15:47 PM
rahul24 joined the channel
2020-01-27 02723, 2020
15:55 PM
rahul24 has quit
2020-01-27 02729, 2020
16:00 PM
rahul24 joined the channel
2020-01-27 02721, 2020
16:05 PM
prabal joined the channel
2020-01-27 02714, 2020
16:13 PM
sarthak_jain has quit
2020-01-27 02710, 2020
16:15 PM
rahul24 has quit
2020-01-27 02727, 2020
16:16 PM
reosarevok
