topic HDFSPut in Talend Studio

HDFSPut

Anonymous — Sat, 16 Nov 2024 10:55:47 GMT

I have a environment with a cloudera hadoop vmware server and a windows laptop with talend open studio. I want to copy files into HDFS from my windows environment and have tried to use the HDFSput component. The connection works fine, and im able to browser the HDFS from my workstation. But when I try to copy in files I get this error :
connecting to socket on port 4024
connected
: org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in component tHDFSPut_1
java.io.IOException: DataStreamer Exception:
    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:708)
Caused by: java.nio.channels.UnresolvedAddressException
    at sun.nio.ch.Net.checkAddress(Unknown Source)
    at sun.nio.ch.SocketChannelImpl.connect(Unknown Source)
    at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192)
    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530)
    at org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1622)
    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1420)
    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1373)
    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutp
Do I miss any library files on my workstation, or is it not possible to copy files to the HDFS from my workstaion.
Br
Petter

Re: HDFSPut

Anonymous — Sat, 28 Nov 2015 21:16:59 GMT

I chaged the port number in my connection from 8020 to 50010, and the file is created but it's empty, here are the output
connecting to socket on port 3993
connected
: org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in component tHDFSPut_1
java.io.EOFException: End of File Exception between local host is: "NOHUSEBPET01/192.168.136.1"; destination host is: "192.168.181.128":50010; : java.io.EOFException; For more details see:
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
    at java.lang.reflect.Constructor.newInstance(Unknown Source)
    at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:791)
    at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764)
    at org.apache.hadoop.ipc.Client.call(Client.java:1472)
    at org.apache.hadoop.ipc.Client.call(Client.java:1399)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
    at com.sun.proxy.$Proxy7.mkdirs(Unknown Source)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.mkdirs(ClientNamenodeProtocolTranslatorPB.java:539)
disconnected
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
    at com.sun.proxy.$Proxy8.mkdirs(Unknown Source)
    at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:2758)
    at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2729)
    at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:870)
    at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:866)
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirsInternal(DistributedFileSystem.java:866)
    at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:859)
    at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1817)
    at local_project.uu_0_1.uu.tHDFSPut_1Process(uu.java:442)
    at local_project.uu_0_1.uu.runJobInTOS(uu.java:819)
    at local_project.uu_0_1.uu.main(uu.java:664)
Caused by: java.io.EOFException
    at java.io.DataInputStream.readInt(Unknown Source)
    at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1071)
    at org.apache.hadoop.ipc.Client$Connection.run(Client.java:966)
Job uu ended at 22:16 28/11/2015.

Re: HDFSPut

Anonymous — Wed, 30 Dec 2015 10:27:52 GMT

Is this solved ?

Re: HDFSPut

Anonymous — Mon, 04 Jan 2016 12:16:57 GMT

Hello,
Can you confirm that the machine you are running the job on can access the machine "192.168.181.128" on the port 50010 ? It looks to be a firewall issue. Also, can you double check that the machine your are running the job on can ping "192.168.181.128" ?

Re: HDFSPut

_AnonymousUser — Fri, 15 Jan 2016 12:50:55 GMT

I have the same issue using hortonworks instead cloudera, and yes the port 50010 is open, all ports have been opened in the firewall, and I can make a telnet to that port.

Re: HDFSPut

Anonymous — Mon, 16 May 2016 20:40:48 GMT

Was this solved? i have the same issue.

Re: HDFSPut

Anonymous — Fri, 27 May 2016 01:12:28 GMT

Check to see if the user account you are using has write permission to the folder you are writing.

Re: HDFSPut

Anonymous — Mon, 30 May 2016 08:00:22 GMT

i have the same issue
Exception in component tHDFSPut_1
java.io.IOException: DataStreamer Exception:
: org.apache.hadoop.hdfs.DFSClient - DataStreamer Exception
java.nio.channels.UnresolvedAddressException
at sun.nio.ch.Net.checkAddress(Net.java:101)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:622)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530)
at org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1752)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1530)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1483)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:668)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:796)
Caused by: java.nio.channels.UnresolvedAddressException
at sun.nio.ch.Net.checkAddress(Net.java:101)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:622)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530)
at org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1752)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1530)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1483)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:668)

Re: HDFSPut

Anonymous — Tue, 07 Jun 2016 15:16:50 GMT

Hi yepeng2007fei, your issue may be caused by this....
https://community.talend.com/t5/Design-and-Development/resolved-Simple-tHDFSPut-based-job-fails-on-6-1-but-not-6-0/td-p/103608