#metabrainz

i unlinked and relinked spotify account on it yesterday also

console paste: https://paste.ee/p/jTCqV

The last line looks not-so-good: "SyntaxError: class is a reserved identifier"

Yeah. CatQuest are you on Firefox < 45?

yea..

I'm working on it!

but like

i know features aren't used

but atleast be able to see regular stuff should be ale, right?

We're compiling the javascript for older browsers too, but it may not include that far…

its not *that* anchient come on :P

That may be an easy change. However we could potentially run into other issues with older browsers.

true

i'm sorry

Ah, actually, never mind. We're not compiling as conservatively as I thought.

uhm

iliekcomputers: could you please run a single query without specifying any drivers or executors?

pristine_: hi.

took 2m20s

same as the rest

which is fast enough rn, imo

ruaok: at this point, i'd prefer to just move ahead with building features.

okay :)

let's read up on tuning stuff in the meanwhile and hopefully, we'll be better equipped to handle scale issues when they come up?

okay :)

iliekcomputers: I agree. but I think pristine_ is doing some great research to understand some of the basic issues.

I am just curious to know about the intricacies of the spark, I will carrry on the research meanwhile we write the queries.

I would love for pristine_ to continue for a bit. but then, lets get user stats working, live updating...

and then.. finally recommendations.

yuss!!

So queries today?

cool :)

ruaok: I will continue :)

I can see something like this being useful for us in the future: https://github.com/lemire/simdjson

planning on dealing with hundreds of gb of data? :D

i have PTSD from watching the last.fm folks trying to scale up. And their peak listens/minute rate still scares the fuck out of me.

how much was it?

too many

how much data is too much data? :P

iliekcomputers: do you want to look at docker/docker-compose on jenkins?

> When I left Last.fm at peak scrobnble time which was about 8-9PM GMT (when Europe was still awake and the US was getting into the swing of things) we had around 600 scrobbles per second coming in.

600/s !!

that is a lot of data

our nginx gateway is currently handling 1000 req/s

ruaok: horizontal scalability, and special attention to SPoF removal

alastairp: hi.

doesn't jenkins use the docker compose from the repository

Zas: remember our recent conversation about this in BCN?

Have you seen the number of listenbrainz containers we already have?

iliekcomputers: a docker-compose config file, you mean?

or the binary?

i meant config

yes, I believe so

https://github.com/metabrainz/acousticbrainz-se... do we just run this script, or is this copied into the jenkins "run this script" box?

copied into the jenkins run this script box.

error while running on all data (pasting here for myself) https://www.irccloud.com/pastebin/qeFDl2Ag/

right. and inside the jenkins container we have this output: Docker environment: / Docker version 1.12.3, build 6b644ec / docker-compose version 1.23.2, build 1110ad0

which is quite old

_except_

I think it's true that we bind-mount the docker socket from that container's host into the jenkins container, so perhaps it's actually dealing with a server which is much more up-to-date

what host is jenkins running on, and what are its versions of docker/d-c?

alastairp: let me look.

and what's the process to rebuild / update the jenkins container? What other things do we test on it, and how far out of date is it?

alastairp: jenkins is on williams

metabrainz/picard#4351 (master - ed28363 : Philipp Wolfer): The build passed.

Change view : https://github.com/metabrainz/picard/compare/d9...

Build details : https://travis-ci.org/metabrainz/picard/builds/...

the version number of jenkins in the footer says 2.150, which is from late-2018, so I don't see why we have to upgrade jenkins itself

right, and can you confirm about the bind mount?

one sec

do you know where the build scripts for jenkins are? (or do we use off-the-shelf images from jenkins?)

https://github.com/metabrainz/jenkins/blob/mast... mmm, looks like this is it

zas: I think I have found a rather easy way to fix PICARD-1178. I would like to include this in a 2.1.3 release, but we could already ask for translations

PICARD-1178: Images tagged with extra types that the user has chosen to ignore should not be shown as 'modified' https://tickets.metabrainz.org/browse/PICARD-1178

zas: what was your plan regarding this? did you want to do an actual dev release + blog post?

phw/picard#111 (master - a10c7a0 : Philipp Wolfer): The build passed.

Change view : https://github.com/phw/picard/compare/f148cd6e7...

Build details : https://travis-ci.org/phw/picard/builds/496478318

bitmap: did you push a new version of jenkins a few weeks ago (while trying to make tests work?)

everyone! today is my 15th MusicBrainz anniversary!

CatQuest: <3

iliekcomputers: :D

CatQuest: Happy Anniversary:)

CatQuest: i'll buy cake to celebrate

where is the party?

Ô_Ô

cake sounds good.

let see if the locals believe in that.

metabrainz/picard#4353 (master - b4664f6 : Philipp Wolfer): The build passed.

Change view : https://github.com/metabrainz/picard/compare/ed...

Build details : https://travis-ci.org/metabrainz/picard/builds/...

iliekcomputers: Just pulled batch-queries from playground.

the point is...

If we collect all the data by running a single query then

we will have to run a for loop to segregate data using pythonic ways

whilst we run the for loop the data would be in driver's memory. So we need to set driver's memory keeping in mind the loop because

it may happen that 1GB memory is enough for .collect but not enough for saving dicts whilst the loop runs

ok I'm trying to understand wtf is blockchain and I came across https://static.blockchain.wtf/wp-content/upload...

and i'm like .. "This sounds like torrenting"

¯\_(ツ)_/¯

pristine_: right.

we'll have to look at avoiding storing data in memory.

pft

lol.

ruaok: we should store listens in the blockchain

I am so over the blockchain it is painful.

it is web-scale

I *still* don't udnerstand wtf it's suppsoed to be though

I'm coming to slap you iliekcomputers!

ooh, a visit from the boss

( ͡° ͜ʖ ͡°)

oh, awww :(

now, public ledgers are really cool. never mind that technically they form the heart of a blockchain.

but today no one can keep blockchain separate from cryptocurrency

iliekcomputers: but

ack. community is slow af now

zas^

the dict size would never be greater than the data collected by .collect because that is what we are storing.

So I think we don't need to worry about that.

Lol

pristine_: wokay.

sounds cool.

metabrainz/picard#4354 (master - a10c7a0 : Philipp Wolfer): The build passed.

Change view : https://github.com/metabrainz/picard/compare/b4...

Build details : https://travis-ci.org/metabrainz/picard/builds/...