#metabrainz

zas: no wifi on train, I take it?

just boarded, intermittent wifi and/or 4G

perfect for doing remote work! :) :)

woo!

Leo_Verto: so, no luck on that problem from yesterday.

hm

and even with the firewall completely turned off, it doesn't work.

can you access hadoop from another container in spark-network?

yep

okay, are you sure setting up the services on a dedicated overlay network is what you want to do?

yes, yes.

are for a login?

care

sure

which key should I use?

and login?

if I wanted to make a service externally accessable I'd use a bridged/host network instead, not sure if that works with services though

although I use expose instead of publish, hm

the overlay network is what enabled the nodes to easily communicate with one another.

which is the whole reason why I am using docker swarm.

and it works quite well from what I can see. except this whole publishing ports bit

yeah, I think you might need an additional bridged network for publish to work

at least that's how I got my (non-swarm) setup working

hmm.

so, create bridged network, add services to both bridged network and to overlay network.

and then?

I mean that's how it'd work with normal containers, not entirely sure what the ingress network does here

also my network graphing tool is broken, nice

except the it works for port 8080, for the spark-master service.

wget http://localhost:8080 -> works great.

that goes into the spark service. but the hadoop service one doesn'twork.

and they are both on the overlay network.

if you docker inspect the individual containers you can see that not only are they on spark-network but also ingress

thats normal/correct, no?

yeah, I'm pretty sure spark-network isn't really used here, the requests use ingress

and running docker service inspect --format="{{json .Endpoint.Spec.Ports}}" <service> I get pretty much the same setup for both

for a single node setup it isn't used, but won't it be needed for a multi-node setup (which we use)?

lateron definitely, but not right now. maybe we should try removing it to see if it interferes here?

sure.

would you like to try it yourself or do you want me to make the changes?

if you clone this, you can have a play yourself

any reason you aren't using compose? :P

compose doesn't do swarms. or at least it didn't when I started this mess.

and I'm not interested in using compose much when I have to reinvent the wheel into normal run commands half the time. might as well just do run commands from the get go.

I just pushed the publish change from last night.

but if you clone this and then run docker/start-master-service.sh

you can start it yourself.

okay, thanks

then start-worker-service.sh

and the requisite stop scripts.

a working setup should yield both 8088 and 8080 services being mapped to localhost.

okay these containers take forever to pull

ruaok, figured it out

the webinterface on hadoop-master is binding to 10.0.0.10 for some reason

and 10.0.0.0/24 is spark-network

I think setting "yarn.nodemanager.runtime.linux.docker.default-container-network" and "yarn.nodemanager.runtime.linux.docker.allowed-container-networks" in the yarn site config here https://github.com/metabrainz/hadoop-cluster-do... should fix it

That seems ok, no?

there's this doc entry about running hadoop in docker https://hadoop.apache.org/docs/current/hadoop-y...

well, that means you can't access the webinterface from outside spark-network, no matter what you publish

ah, the publish command doesn't bridge it?

nope, the container has to be on a bridge network and the service inside must bind to that network or all of them

ok, let me try the settings you suggested.

but the ingress network should work as a bridge, I think

that is 10.255.0.0/24 btw

yarn.nodemanager.runtime.linux.docker.default-container-network ==> ingress

?

yeah

like this perhaps?

hmm, reading more of that article it seems like that requires using the "Linux Container Executor" to start yarn which is not something we want, right?

dunno what that exactly is. I want swarm to start the containers.

yeah, th

there has to be a simpler way of just changing the bind address

but for some reason the documentation doesn't mention the web server, ugh https://mapr.com/docs/home/ReferenceGuide/yarn-...

and we're on hadoop 3.1.1. at that...

oh yeah

yarn.resourcemanager.hostname

apparently that sort of stuff is "documented" here https://hadoop.apache.org/docs/r3.1.1/hadoop-ya...

so you should be fine setting that to 0.0.0.0 I suppose, otherwise it wouldn't be accessible from spark-network

Let me that that in a bit, afk for a moment.

iliekcomputers, you should see New Year's Eve here. one year we had so much smog there was only ~15m visibility

So dumb.

literally lights up the particulates sensor maps https://www.umweltbundesamt.de/sites/default/fi...

Official fireworks time is supposed to be 8-10 PM

There's official firewords time? :D

Hi Soumya

welcome to metabrainz irc

Greetings! I need some help regarding my task..

which task is it?

I have to collect some imteresting facts aboit METABRAINZ

Interesting***

About***

Soumya: Hi, can you link the task you're working on?

reosarevok: there is! it's to stop them going all night I guess, but nobody cares ;)

https://codein.withgoogle.com/dashboard/task-in.... This is the link

Soumya: ok, thanks. I can talk to you about CritiqueBrainz, ListenBrainz or AcousticBrainz if you want. I'm not a mentor of that task, so I don't know exactly what kind of facts they're expecting.

Okay.... I'll ask after a few minutes of researching...

Soumya, also, non-mentors can't follow that link, but might have interesting input. Can you pastebin the task description?

CallerNo6: https://www.irccloud.com/pastebin/oz9QwZwM/

iliekcomputers, thanks!

=)

uploading dumps to hdfs! :D https://usercontent.irccloud-cdn.com/file/0tcUZ...

on your server?

on my laptop, local small dump :D

k.

perfect timing. because I hope to have hdfs up and running version soon

Leo_Verto: that 0.0.0.0 trick worked and in hindsight makes total sense.

yeah, all the time spent digging through docker docs and it's something as simple as binding to the wrong interface…

yeah...

anyway, let me know when spark's running and we can get started with the jupyter notebook server :D

looks like it needs more config settings.

yarn.resourcemanager.hostname defines the hostname which others refer to the service with.

useless for connecting to the service.

yarn.resourcemanager.bind-host I bet.

[picard] phw opened pull request #1027: PICARD-1375: Handle empty values in metadata sanitation (master...PICARD-1375) https://git.io/fpkkR

iliekcomputers: exciting, I like that PR!

ruaok: yusss. =)

For streaming I was thinking, a simple flask app with an API and a spark writer on one of the nodes

please elaborate on that thought -- sounds complicated.

the spark nodes will probably not have a direct connection with lemmy right?

run a small flask app on the nodes to which the `biquery-writer` equivalent in listenbrainz submit new listens

the flask app then submits to a script running on the nodes which writes them to hdfs

> the spark nodes will probably not have a direct connection with lemmy right?

they could

rabbitmq makes a good isolator in case the cluster is not available.

could the spark nodes access rabbitmq?

yes.