#metabrainz

so, what have you learned, iliekcomputers?

ruaok: so i was mucking around with hadoop and config values yesterday.

there's two interfaces that clients can use to read/write/interact with hdfs

one is the standard rpc interface, the other is a http api type interface called webhdfs

do you have a feeling for which is preferred?

ruaok: most of the simpler python clients I've looked at use webhdfs

the client I wrote the code with uses webhdfs too

`hdfs dfs -put` used RPC

and that is the one interface we got working so far. lol.

yeah, about that.

i changed some config values and used `stop-master-service` and `start-master-service`

and now the put thing isn't working...

heh, lol.

i'm not sure why exactly

hang on, there might be uncommited changes in my repo.

the latest hadoop-yarn image is just the master branch of github/meb/hadoop-docker

the playground repo only has these two lines

+ -p published=8088,target=8088,mode=host \

+ -p published=9864,target=9864 \

which don't affect anything.

maybe it's the worker nodes. how exactly do I find out what IP they are on and if they're running.

from the PR Leo_Verto made, it seemed like `hdfs dfsadmin -report` wasn't trustworthy.

changes commited, and yes those would make a difference.

I've built a new image and am pushing it now

push.

+ed

ok cool

you should be able to stop/start and have it work again.

let me try the put thing again.

:| https://www.irccloud.com/pastebin/EtM0dvNR/

did you restart both the master and workers?

i don't have access to the workers.

you don't need it.

afaik

./stop-workers.sh

ah

./start-workser.sh

oh cool

-s

the nice thing bout this system is that you never have to care about the worker nodes. they get fired up and join the cluster.

Usage: start-worker-service.sh <replicas>

replicas is a number?

3?

yep

(to both)

noice, that's pretty cool.

still the same error

can you confirm for once that the workers are on 10.0.0.24, 10.0.0.23 etc?

the 10 network is the overlay network and every time you start and stop the workers, they get new IPs.

I'm going to restart the cluster using my setup. hang on

now it works again.

why?

ruaok: foo is an empty file

yea, I just touched it.

once it has data, it doesn't work...

try again, i echoed some text into it.

doh.

huh, interesting.

but that used to work, no?

well, we should really focus on the http version of this interface and go with that.

are you annoyed by the fact that the only error messages that hadoop gives are java stacktraces?

yes.

not very helpful

we'd get flayed alive if we wrote code like that.

but int he java world, I guess you're used to pain.

I think we should look at what ports are accessible to the master from the workers

let me log into a worker

there's probably some reason why the master isn't able to connect with the workers

the workers have quite fewer ports open.

The ports 9866 and 9864 are the ones we want accessible from the master for sure

as in a process on master makes a connection to port 9866/9864 on a worker?

ruaok: yes

ok, both of those are exposed on the inside of the worker containers.

could you ping 10.0.0.39 ?

ping works

ok, then we need to find which setting causes those ports to be bound on all interfaces.

dfs.namenode.http-bind-host ?

yay!

both ports are not reachable from the master node.

*now

I'm still getting connection refuse from inside the master container

ok, remember that hadoop-master is the master node.

03a472b53968 is a worker node

try wget http://03a472b53968:9866

no data received. but it connected.

probably an invalid wget command, but at least the port is open.

yes

I see that the client is trying to connect to incorrect IPs.

This was the thing that Leo_Verto had fixed

did we lose his PR somehow?

core-site.xml in /usr/local/hadoop/etc/hadoop/ has the config value set correctly.

maybe it needs to be in hdfs-site.xml ?

can't hurt to try

change applied, cluster restarted.

try it again, iliekcomputers

two workers errored but I think it worked

I wonder if that is because we just restarted things...

Hi CatQuest

it is weird.

from the logs it still seems like it can't connect to 2 of the datanodes.

just got a long list of gamelan set (it's 26) from my Javanese Art teacher :)

which ones?

remember that you can look at which ones are alive here:

can you add dfs.datanode.use.datanode.hostname to true once too?

once?

I didn't parse what you're requesting

whoops

can you set dfs.datanode.use.datanode.hostname to true in hdfs-site.xml?

and then I'll try again.

done

no error this time, woo!

now let's check if the http interface works...