#metabrainz

i can do such that :)

[picard] phw opened pull request #977: i18n: Enable languages with more than 40% translation (master...enable-more-languages) https://git.io/fA7XF

metabrainz/picard#3777 (master - f108cc7 : Philipp Wolfer): The build passed.

Change view : https://github.com/metabrainz/picard/compare/eb...

Build details : https://travis-ci.org/metabrainz/picard/builds/...

metabrainz/picard#3777 (master - f108cc7 : Philipp Wolfer): The build passed.

Change view : https://github.com/metabrainz/picard/compare/eb...

Build details : https://travis-ci.org/metabrainz/picard/builds/...

bitmap, yvanzo: high number of 500s on wetton, ludwig, hip, started on 2018-09-21, 00:30 UTC, i restarted mb ws containers on those

last message before it starts (from ludwig & hip logs, same message) https://www.irccloud.com/pastebin/PjZvfZon/

ok, now coffee ;)

nom. :)

iliekcomputers: what is the blue jelly? (i assume red and yellow strawberry/raspberry and is pineapple/lemon) but what is the blue?

orwait it's probably blue raspberry :C

ruaok: hey, let's catch up on LB when you want

k, let's do it.

ok, high level view.

stats are broken, we have a messy spotify patch looming and we need to replay some data.

what other short term projects are there?

can someone bring a video projector for the summit ? It'd be convenient for small presentations

We had that last year, right? Where did we get it from?

I mean, we definitely had stuff on a screen because I remember Leo_Verto presenting some results

Unless I'm going mad, that is

I don't dispute that.

I just can't remember.

But the office itself doesn't have a projector? :)

I've got one I can bring

Oh, that makes it easy :)

Yes and I hope we can live stream too like last year

Mr_Monkey: perfect

ruaok: Spotify thing should be deployed first

It's done, we decided on waiting for better logging but i fixed that after the BQ bill...

Other than that, Spark setup is something we should discuss

agreed, I want to get spotify out of the way -- so, do that first.

and yes on spark. I've been trying to wrap my head around what we need to do.

Okay, Spotify on beta for testing

ok, good first step. there were tons of open tickets related to that -- have you churned through all those already?

ruaok: yeah

huh. there is a docker event happening on dec 3 here in BCN.

Most of them were fixed in the main PR

I'll check again once

and apparently my old college roommate is organizing it and just invited me. so odd.

beta testing is going to be.. tricky. I think.

we dont have 'throwaway spotify accounts' and it is a service that needs to keep running.

let me think about a good way to do that, but for now let's move on to spark.

Okay

so, you reworked the images to include... HDFS support, right?

Was working on that, I'll be done with it soon.

ok.

so, from what I see we need to create a namenode and one or more datanotes to create an HPFS networked file system.

does that live on the same node at spark workers/master?

or do they need separate instances?

I was thinking they'd be on the same nodes

lb is pretty small rn, I'm not sure if keeping seperate nodes for storage is needed.

certainly sounds not too difficult to change.

let's plan on having a master node that has spark master and namenode.

and spark workers that have datanodes.

1-to-1 mapping.

I think that should be a safe starting point.

Okay, sounds good to me.

I've been looking at various project that seem to do this on github. there are about 3 of them, most of them have had zero commits in the last year.

where this is... HDFS in docker.

none seem to be a an out of the box fit -- I think we should take one that looks reasonable fork it and tweak it to our needs.

SSH passwordless access and installing packages and running the nodes seem to be the biggest tasks.

then we need to load and update the LB data into the cluster -- I think those might be two distinct tasks.

finally, then we need to bolt the stats back on top.

once that is working we can get back to the recommendation engine.

that feels like the right path for me, what do you think?

Okay, these steps sound good.

I'd say one to 1.5 weeks for each task should be possible.

I'm looking at my calendar and really... ugh. I doubt I'll have a lot of time to work on this stuff until mid-oct.

summit, beginning of quarter, travel.

Right. Forgot about the travel

maybe on friday we could spend some time working on this stuff together.

Okay

like getting a HDFS cluster up and running.

I'll hack on the Dockerfile in the meanwhile

And see what needs to be done with spotify

ok, sounds good.

I was looking at these projects: https://github.com/sequenceiq/hadoop-docker

there was another, but I think the latter one looks like a decent starting point.

Okay, I've seen the first one. I'll look into the second one.

ruaok: hi, does spike have a domain name?