10:48 AM
CatQuest
i unlinked and relinked spotify account on it yesterday also
2019-02-21 05254, 2019
10:48 AM
CatQuest
2019-02-21 05252, 2019
10:49 AM
Mr_Monkey
The last line looks not-so-good: "SyntaxError: class is a reserved identifier"
2019-02-21 05249, 2019
10:50 AM
Mr_Monkey
Yeah. CatQuest are you on Firefox < 45?
2019-02-21 05210, 2019
10:51 AM
CatQuest
yea..
2019-02-21 05214, 2019
10:51 AM
CatQuest
I'm working on it!
2019-02-21 05217, 2019
10:51 AM
CatQuest
but like
2019-02-21 05214, 2019
10:52 AM
CatQuest
i know features aren't used
2019-02-21 05225, 2019
10:52 AM
CatQuest
but atleast be able to see regular stuff should be ale, right?
2019-02-21 05233, 2019
10:52 AM
Mr_Monkey
We're compiling the javascript for older browsers too, but it may not include that far…
2019-02-21 05255, 2019
10:52 AM
CatQuest
its not *that* anchient come on :P
2019-02-21 05212, 2019
10:53 AM
Mr_Monkey
That may be an easy change. However we could potentially run into other issues with older browsers.
2019-02-21 05240, 2019
10:53 AM
CatQuest
true
2019-02-21 05245, 2019
10:53 AM
CatQuest
i'm sorry
2019-02-21 05202, 2019
10:54 AM
Mr_Monkey
Ah, actually, never mind. We're not compiling as conservatively as I thought.
2019-02-21 05210, 2019
10:54 AM
pulkit6559 has quit
2019-02-21 05213, 2019
10:54 AM
CatQuest
uhm
2019-02-21 05216, 2019
10:54 AM
pristine_
iliekcomputers: could you please run a single query without specifying any drivers or executors?
2019-02-21 05233, 2019
10:54 AM
CatQuest goes of to have breakfast
2019-02-21 05243, 2019
10:55 AM
iliekcomputers
pristine_: hi.
2019-02-21 05205, 2019
10:57 AM
iliekcomputers
took 2m20s
2019-02-21 05208, 2019
10:57 AM
iliekcomputers
same as the rest
2019-02-21 05221, 2019
10:57 AM
iliekcomputers
which is fast enough rn, imo
2019-02-21 05230, 2019
10:57 AM
ruaok agrees
2019-02-21 05235, 2019
10:57 AM
iliekcomputers
ruaok: at this point, i'd prefer to just move ahead with building features.
2019-02-21 05256, 2019
10:57 AM
pristine_
okay :)
2019-02-21 05223, 2019
10:58 AM
iliekcomputers
let's read up on tuning stuff in the meanwhile and hopefully, we'll be better equipped to handle scale issues when they come up?
2019-02-21 05237, 2019
10:58 AM
pristine_
okay :)
2019-02-21 05227, 2019
10:59 AM
ruaok
iliekcomputers: I agree. but I think pristine_ is doing some great research to understand some of the basic issues.
2019-02-21 05240, 2019
10:59 AM
pristine_
I am just curious to know about the intricacies of the spark, I will carrry on the research meanwhile we write the queries.
2019-02-21 05251, 2019
10:59 AM
ruaok
I would love for pristine_ to continue for a bit. but then, lets get user stats working, live updating...
2019-02-21 05256, 2019
10:59 AM
ruaok
and then.. finally recommendations.
2019-02-21 05204, 2019
11:00 AM
iliekcomputers
yuss!!
2019-02-21 05204, 2019
11:00 AM
pristine_
So queries today?
2019-02-21 05257, 2019
11:02 AM
pristine_
cool :)
2019-02-21 05218, 2019
11:08 AM
pristine_
ruaok: I will continue :)
2019-02-21 05243, 2019
11:11 AM
ruaok
grand.
2019-02-21 05258, 2019
11:17 AM
ruaok
2019-02-21 05200, 2019
11:18 AM
pulkit6559 joined the channel
2019-02-21 05253, 2019
11:20 AM
madmouser1 has quit
2019-02-21 05259, 2019
11:30 AM
alastairp
planning on dealing with hundreds of gb of data? :D
2019-02-21 05230, 2019
11:33 AM
pulkit6559 has quit
2019-02-21 05231, 2019
11:34 AM
ruaok
i have PTSD from watching the last.fm folks trying to scale up. And their peak listens/minute rate still scares the fuck out of me.
2019-02-21 05249, 2019
11:49 AM
pulkit6559 joined the channel
2019-02-21 05243, 2019
12:04 PM
iliekcomputers
how much was it?
2019-02-21 05224, 2019
12:11 PM
alastairp
too many
2019-02-21 05245, 2019
12:11 PM
iliekcomputers
how much data is too much data? :P
2019-02-21 05220, 2019
12:14 PM
alastairp
iliekcomputers: do you want to look at docker/docker-compose on jenkins?
2019-02-21 05205, 2019
12:15 PM
ruaok
> When I left Last.fm at peak scrobnble time which was about 8-9PM GMT (when Europe was still awake and the US was getting into the swing of things) we had around 600 scrobbles per second coming in.
2019-02-21 05218, 2019
12:15 PM
ruaok
600/s !!
2019-02-21 05212, 2019
12:22 PM
pprkut has quit
2019-02-21 05219, 2019
12:22 PM
pprkut joined the channel
2019-02-21 05204, 2019
12:23 PM
Tykling
that is a lot of data
2019-02-21 05233, 2019
12:24 PM
zas
our nginx gateway is currently handling 1000 req/s
2019-02-21 05209, 2019
12:25 PM
zas
ruaok: horizontal scalability, and special attention to SPoF removal
2019-02-21 05202, 2019
12:30 PM
iliekcomputers
alastairp: hi.
2019-02-21 05214, 2019
12:30 PM
iliekcomputers
doesn't jenkins use the docker compose from the repository
2019-02-21 05219, 2019
12:30 PM
ruaok
Zas: remember our recent conversation about this in BCN?
2019-02-21 05257, 2019
12:30 PM
ruaok
Have you seen the number of listenbrainz containers we already have?
2019-02-21 05228, 2019
12:37 PM
alastairp
iliekcomputers: a docker-compose config file, you mean?
2019-02-21 05231, 2019
12:37 PM
alastairp
or the binary?
2019-02-21 05242, 2019
12:37 PM
iliekcomputers
i meant config
2019-02-21 05253, 2019
12:37 PM
alastairp
yes, I believe so
2019-02-21 05203, 2019
12:41 PM
alastairp
2019-02-21 05231, 2019
12:41 PM
iliekcomputers
copied into the jenkins run this script box.
2019-02-21 05249, 2019
12:42 PM
iliekcomputers
2019-02-21 05225, 2019
12:43 PM
alastairp
right. and inside the jenkins container we have this output: Docker environment: / Docker version 1.12.3, build 6b644ec / docker-compose version 1.23.2, build 1110ad0
2019-02-21 05249, 2019
12:43 PM
alastairp
which is quite old
2019-02-21 05252, 2019
12:43 PM
alastairp
_except_
2019-02-21 05231, 2019
12:44 PM
alastairp
I think it's true that we bind-mount the docker socket from that container's host into the jenkins container, so perhaps it's actually dealing with a server which is much more up-to-date
2019-02-21 05241, 2019
12:44 PM
Lotheric_ is now known as Lotheric
2019-02-21 05247, 2019
12:44 PM
alastairp
what host is jenkins running on, and what are its versions of docker/d-c?
2019-02-21 05258, 2019
12:44 PM
iliekcomputers
alastairp: let me look.
2019-02-21 05224, 2019
12:45 PM
alastairp
and what's the process to rebuild / update the jenkins container? What other things do we test on it, and how far out of date is it?
2019-02-21 05201, 2019
12:47 PM
iliekcomputers
alastairp: jenkins is on williams
2019-02-21 05220, 2019
12:47 PM
travis-ci joined the channel
2019-02-21 05222, 2019
12:47 PM
travis-ci
metabrainz/picard#4351 (master - ed28363 : Philipp Wolfer): The build passed.
2019-02-21 05222, 2019
12:47 PM
travis-ci
2019-02-21 05222, 2019
12:47 PM
travis-ci
2019-02-21 05222, 2019
12:47 PM
travis-ci has left the channel
2019-02-21 05222, 2019
12:47 PM
alastairp
the version number of jenkins in the footer says 2.150, which is from late-2018, so I don't see why we have to upgrade jenkins itself
2019-02-21 05223, 2019
12:47 PM
iliekcomputers
2019-02-21 05237, 2019
12:47 PM
alastairp
right, and can you confirm about the bind mount?
2019-02-21 05246, 2019
12:47 PM
iliekcomputers
2019-02-21 05250, 2019
12:47 PM
iliekcomputers
one sec
2019-02-21 05207, 2019
12:48 PM
alastairp
do you know where the build scripts for jenkins are? (or do we use off-the-shelf images from jenkins?)
2019-02-21 05253, 2019
12:48 PM
alastairp
2019-02-21 05204, 2019
12:49 PM
outsidecontext
zas: I think I have found a rather easy way to fix PICARD-1178. I would like to include this in a 2.1.3 release, but we could already ask for translations
2019-02-21 05204, 2019
12:49 PM
BrainzBot
2019-02-21 05232, 2019
12:49 PM
outsidecontext
zas: what was your plan regarding this? did you want to do an actual dev release + blog post?
2019-02-21 05208, 2019
12:50 PM
travis-ci joined the channel
2019-02-21 05209, 2019
12:50 PM
travis-ci
phw/picard#111 (master - a10c7a0 : Philipp Wolfer): The build passed.
2019-02-21 05209, 2019
12:50 PM
travis-ci
2019-02-21 05209, 2019
12:50 PM
travis-ci
2019-02-21 05209, 2019
12:50 PM
travis-ci has left the channel
2019-02-21 05254, 2019
12:53 PM
D4RK-PH0ENiX has quit
2019-02-21 05217, 2019
12:57 PM
alastairp
bitmap: did you push a new version of jenkins a few weeks ago (while trying to make tests work?)
2019-02-21 05255, 2019
12:58 PM
CatQuest
everyone! today is my 15th MusicBrainz anniversary!
2019-02-21 05221, 2019
12:59 PM
iliekcomputers
CatQuest: <3
2019-02-21 05238, 2019
12:59 PM
CatQuest
iliekcomputers: :D
2019-02-21 05259, 2019
12:59 PM
pristine_
CatQuest: Happy Anniversary:)
2019-02-21 05202, 2019
13:00 PM
iliekcomputers
CatQuest: i'll buy cake to celebrate
2019-02-21 05217, 2019
13:00 PM
pristine_
where is the party?
2019-02-21 05229, 2019
13:00 PM
CatQuest
Ô_Ô
2019-02-21 05240, 2019
13:04 PM
ruaok
cake sounds good.
2019-02-21 05249, 2019
13:04 PM
ruaok
let see if the locals believe in that.
2019-02-21 05202, 2019
13:07 PM
travis-ci joined the channel
2019-02-21 05203, 2019
13:07 PM
travis-ci
metabrainz/picard#4353 (master - b4664f6 : Philipp Wolfer): The build passed.
2019-02-21 05203, 2019
13:07 PM
travis-ci
2019-02-21 05203, 2019
13:07 PM
travis-ci
2019-02-21 05203, 2019
13:07 PM
travis-ci has left the channel
2019-02-21 05206, 2019
13:07 PM
pristine_
iliekcomputers: Just pulled batch-queries from playground.
2019-02-21 05250, 2019
13:07 PM
pristine_
the point is...
2019-02-21 05238, 2019
13:08 PM
pristine_
If we collect all the data by running a single query then
2019-02-21 05216, 2019
13:09 PM
pristine_
we will have to run a for loop to segregate data using pythonic ways
2019-02-21 05225, 2019
13:10 PM
pristine_
whilst we run the for loop the data would be in driver's memory. So we need to set driver's memory keeping in mind the loop because
2019-02-21 05250, 2019
13:11 PM
pristine_
it may happen that 1GB memory is enough for .collect but not enough for saving dicts whilst the loop runs
2019-02-21 05210, 2019
13:12 PM
CatQuest
2019-02-21 05210, 2019
13:12 PM
CatQuest
and i'm like .. "This sounds like torrenting"
2019-02-21 05222, 2019
13:12 PM
CatQuest
¯\_(ツ)_/¯
2019-02-21 05245, 2019
13:12 PM
iliekcomputers
pristine_: right.
2019-02-21 05229, 2019
13:13 PM
iliekcomputers
we'll have to look at avoiding storing data in memory.
2019-02-21 05256, 2019
13:13 PM
ruaok
2019-02-21 05208, 2019
13:14 PM
CatQuest
pft
2019-02-21 05217, 2019
13:14 PM
ruaok
lol.
2019-02-21 05224, 2019
13:14 PM
iliekcomputers
ruaok: we should store listens in the blockchain
2019-02-21 05226, 2019
13:14 PM
ruaok
I am so over the blockchain it is painful.
2019-02-21 05228, 2019
13:14 PM
iliekcomputers
it is web-scale
2019-02-21 05235, 2019
13:14 PM
ruaok looks for tickets to delhi
2019-02-21 05241, 2019
13:14 PM
CatQuest
I *still* don't udnerstand wtf it's suppsoed to be though
2019-02-21 05255, 2019
13:14 PM
ruaok
I'm coming to slap you iliekcomputers!
2019-02-21 05201, 2019
13:15 PM
iliekcomputers
ooh, a visit from the boss
2019-02-21 05203, 2019
13:15 PM
CatQuest
( ͡° ͜ʖ ͡°)
2019-02-21 05205, 2019
13:15 PM
iliekcomputers
oh, awww :(
2019-02-21 05223, 2019
13:16 PM
ruaok
now, public ledgers are really cool. never mind that technically they form the heart of a blockchain.
2019-02-21 05236, 2019
13:16 PM
ruaok
but today no one can keep blockchain separate from cryptocurrency
2019-02-21 05253, 2019
13:16 PM
ruaok goes to read the PRs from Mr_Monkey
2019-02-21 05213, 2019
13:17 PM
pristine_
iliekcomputers: but
2019-02-21 05219, 2019
13:17 PM
CatQuest
ack. community is slow af now
2019-02-21 05228, 2019
13:17 PM
CatQuest
zas^
2019-02-21 05254, 2019
13:17 PM
pristine_
the dict size would never be greater than the data collected by .collect because that is what we are storing.
2019-02-21 05225, 2019
13:18 PM
pristine_
So I think we don't need to worry about that.
2019-02-21 05227, 2019
13:18 PM
pristine_
Lol
2019-02-21 05250, 2019
13:18 PM
madmouser1 joined the channel
2019-02-21 05232, 2019
13:19 PM
iliekcomputers
pristine_: wokay.
2019-02-21 05235, 2019
13:19 PM
iliekcomputers
sounds cool.
2019-02-21 05245, 2019
13:20 PM
travis-ci joined the channel
2019-02-21 05246, 2019
13:20 PM
travis-ci
metabrainz/picard#4354 (master - a10c7a0 : Philipp Wolfer): The build passed.
2019-02-21 05247, 2019
13:20 PM
travis-ci
2019-02-21 05247, 2019
13:20 PM
travis-ci
