Talend and HDFS Connectivity

Anonymous — Fri, 15 Sep 2017 11:24:06 GMT

Hi All,

I am trying to put a file from my local directory to HDFS, but getting the below error.

Starting job m_write_to_hdfs at 15:49 15/09/2017.

[statistics] connecting to socket on port 3504
[statistics] connected
[WARN ]: org.apache.hadoop.util.Shell - Did not find winutils.exe: java.io.FileNotFoundException: java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset. -see https://wiki.apache.org/hadoop/WindowsProblems
[WARN ]: org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in component tHDFSPut_1
java.io.IOException: DataStreamer Exception:
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:577)
Caused by: java.nio.channels.UnresolvedAddressException
at sun.nio.ch.Net.checkAddress(Unknown Source)
at sun.nio.ch.SocketChannelImpl.connect(Unknown Source)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1601)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1342)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1295)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:463)
[WARN ]: org.apache.hadoop.hdfs.DFSClient - DataStreamer Exception
java.nio.channels.UnresolvedAddressException
at sun.nio.ch.Net.checkAddress(Unknown Source)
at sun.nio.ch.SocketChannelImpl.connect(Unknown Source)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1601)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1342)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1295)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:463)
[statistics] disconnected
Job m_write_to_hdfs ended at 15:50 15/09/2017. [exit code=1]

KIndly check if anything needs to be done from my end.

Thanks

Re: Talend and HDFS Connectivity

Anonymous — Mon, 18 Sep 2017 06:52:24 GMT

Hello,

Did you use the hostname of your namenode in your HDFSOutput component? Could you please show us your HDFSOutput component setting screenshot? The machine you are running the job on can access your HDFS connection successfully?

Best regards

Sabrina

Re: Talend and HDFS Connectivity

Anonymous — Thu, 09 Nov 2017 09:45:10 GMT

hi i have the same exception

my namenode uri is:

hdfs://localhost:8020

do i have change anything?

topic Talend and HDFS Connectivity in Talend Studio

Talend and HDFS Connectivity

Re: Talend and HDFS Connectivity

Re: Talend and HDFS Connectivity