topic Re: hdfs connection error in Talend Studio

hdfs connection error

YPMAL — Sat, 16 Nov 2024 07:34:11 GMT

I have installed cloudera VM and using CDH 5.13.

I am able to connect it using putty.

In THdfsInput I can view the HDFS file after choosing browse button. See screentshot view1

But after running I am getting an error

[WARN ]: org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... "using builtin-java classes where applicable
Exception in component tHDFSInput_1 (frstjob)
java.nio.channels.UnresolvedAddressException"

Re: hdfs connection error

manodwhb — Thu, 27 Sep 2018 09:42:02 GMT

@yogeshmalekar,can you un-tick Use Datanode Hostname option in basic settings of tHDFSInput and let me know.

Re: hdfs connection error

YPMAL — Thu, 27 Sep 2018 09:49:39 GMT

After removing use datanode hostname I am getting following error

[statistics] connecting to socket on port 3524
[statistics] connected
[WARN ]: org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[WARN ]: org.apache.hadoop.hdfs.BlockReaderFactory - I/O error constructing remote block reader.
java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530)
at org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:3553)
at org.apache.hadoop.hdfs.BlockReaderFactory.nextTcpPeer(BlockReaderFactory.java:840)
at org.apache.hadoop.hdfs.BlockReaderFa

To see the whole post, download it here
OriginalPost.pdf

Re: hdfs connection error

manodwhb — Thu, 27 Sep 2018 09:57:49 GMT

@yogeshmalekar,revert back that change and can you check that the Namenode URI.

i believe you might have used hdfs://localhost:8020..can you changed to IP address and let me know that is working or not?

Re: hdfs connection error

YPMAL — Thu, 27 Sep 2018 10:29:23 GMT

I am using the ip address. I can connect in putty using that ip address.

Re: hdfs connection error

YPMAL — Thu, 27 Sep 2018 12:25:19 GMT

Thanks.. I was doing two errors.

1) I was not giving port which is 8020 by default

2) Use DataNode HostName was checked.

Thanks for your help.

Re: hdfs connection error

manodwhb — Thu, 27 Sep 2018 13:35:44 GMT

@yogeshmalekar,great,your issue resolved. Kudos also accepted.