Facing error "HADOOP_HOME and hadoop.home.dir are unset" while connecting to HDFS using Talend

[Anonymous]

Hi,

I am trying to connect to HDFS (2.7.3) using Talend big data platform (version 6.3) and loading data from csv to hadoop, but it's throwing below error. Attached is the complete log for the same.

Could you please advise on this. [Note: I also downloaded winutils.exe file (got version 2.6.0 on google) and put it under Talend folder and have also set HADOOP_HOME environment variable but it's not working out]

19:50:30.412 [main] DEBUG org.apache.hadoop.util.Shell - Failed to detect a valid hadoop home directory

java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset.

AND

19:50:30.412 [main] DEBUG org.apache.hadoop.util.Shell - Failed to find winutils.exe

java.io.FileNotFoundException: java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset. -see https://wiki.apache.org/hadoop/WindowsProblems

Appreciate your help!!

Regards,

Pragya

[Anonymous]

Hi,

I downloaded the winutil.exe file from the given URL and set JVM Arguments in advance setting of Talend job like below.
-Dhadoop.home.dir=C:\\winutil\\

So now the above error is resolved, but it has started giving me some different error. below is the same. Do you have any idea about it.


Mar 07, 2018 3:20:09 PM org.apache.hadoop.conf.Configuration warnOnceIfDeprecated
INFO: fs.default.name is deprecated. Instead, use fs.defaultFS
Mar 07, 2018 3:20:09 PM org.apache.hadoop.util.NativeCodeLoader <clinit>
WARNING: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15:20:09.282 [main] DEBUG o.a.hadoop.util.PerformanceAdvisory - Falling back to shell based
Mar 07, 2018 3:20:09 PM org.apache.hadoop.conf.Configuration warnOnceIfDeprecated
INFO: fs.default.name is deprecated. Instead, use fs.defaultFS
15:20:09.333 [main] DEBUG o.a.h.security.UserGroupInformation - PrivilegedAction as:admin (auth IMPLE) from rg.apache.hadoop.fs.FileSystem.get(FileSystem.java:166)
15:20:16.152 [main] DEBUG o.a.hadoop.util.PerformanceAdvisory - Both short-circuit local reads and UNIX domain socket are disabled.
15:20:16.183 [main] DEBUG o.a.h.h.p.d.s.DataTransferSaslUtil - DataTransferProtocol not using SaslPropertiesResolver, no QOP found in configuration for dfs.data.transfer.protection
Mar 07, 2018 3:20:17 PM org.apache.hadoop.io.retry.RetryInvocationHandler handleException
WARNING: Exception while invoking ClientNamenodeProtocolTranslatorPB.mkdirs over null. Not retrying because try once and fail.
java.io.IOException: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.; Host Details : local host is: "01HW1068348/192.168.77.159"; destination host is: "UBUNTU":50070;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:782)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1556)
at org.apache.hadoop.ipc.Client.call(Client.java:1496)
at org.apache.hadoop.ipc.Client.call(Client.java:1396)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
at com.sun.proxy.$Proxy8.mkdirs(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.mkdirs(ClientNamenodeProtocolTranslatorPB.java:603)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[FATAL]: poc_pragya.dataloadtohive_old_0_1.DataLoadToHive_old - tHDFSPut_1 Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.; Host Details : local host is: "01HW1068348/192.168.77.159"; destination host is: "UBUNTU":50070;
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:278)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:194)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:176)
at com.sun.proxy.$Proxy9.mkdirs(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:3061)
at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:3031)
at org.apache.hadoop.hdfs.DistributedFileSystem$24.doCall(DistributedFileSystem.java:1162)
at org.apache.hadoop.hdfs.DistributedFileSystem$24.doCall(DistributedFileSystem.java:1158)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirsInternal(DistributedFileSystem.java:1158)
at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:1150)
at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1898)
at poc_pragya.dataloadtohive_old_0_1.DataLoadToHive_old.tHDFSPut_1Process(DataLoadToHive_old.java:378)
at poc_pragya.dataloadtohive_old_0_1.DataLoadToHive_old.runJobInTOS(DataLoadToHive_old.java:774)
at poc_pragya.dataloadtohive_old_0_1.DataLoadToHive_old.main(DataLoadToHive_old.java:608)
Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.
at com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:94)
at com.google.protobuf.CodedInputStream.checkLastTagWas(CodedInputStream.java:124)
at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:202)
at com.google.protobuf.AbstractParser.parsePartialDelimitedFrom(AbstractParser.java:241)
at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:253)
at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:259)
at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:49)
at org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.parseDelimitedFrom(RpcHeaderProtos.java:4060)
at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1118)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:1012)

Exception in component tHDFSPut_1
java.io.IOException: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.; Host Details : local host is: "01HW1068348/192.168.77.159"; destination host is: "UBUNTU":50070;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:782)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1556)
at org.apache.hadoop.ipc.Client.call(Client.java:1496)
at org.apache.hadoop.ipc.Client.call(Client.java:1396)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
at com.sun.proxy.$Proxy8.mkdirs(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.mkdirs(ClientNamenodeProtocolTranslatorPB.java:603)
[statistics] disconnected
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:278)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:194)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:176)
at com.sun.proxy.$Proxy9.mkdirs(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:3061)
at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:3031)
at org.apache.hadoop.hdfs.DistributedFileSystem$24.doCall(DistributedFileSystem.java:1162)
at org.apache.hadoop.hdfs.DistributedFileSystem$24.doCall(DistributedFileSystem.java:1158)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirsInternal(DistributedFileSystem.java:1158)
at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:1150)
at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1898)
at poc_pragya.dataloadtohive_old_0_1.DataLoadToHive_old.tHDFSPut_1Process(DataLoadToHive_old.java:378)
at poc_pragya.dataloadtohive_old_0_1.DataLoadToHive_old.runJobInTOS(DataLoadToHive_old.java:774)
at poc_pragya.dataloadtohive_old_0_1.DataLoadToHive_old.main(DataLoadToHive_old.java:608)
Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.
at com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:94)
at com.google.protobuf.CodedInputStream.checkLastTagWas(CodedInputStream.java:124)
at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:202)
at com.google.protobuf.AbstractParser.parsePartialDelimitedFrom(AbstractParser.java:241)
at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:253)
at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:259)
at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:49)
at org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.parseDelimitedFrom(RpcHeaderProtos.java:4060)
at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1118)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:1012)

Regards,
Pragya

[Anonymous]

Hello,

From this online user guide:TalendHelpCenter: Supported Hadoop distribution versions for Talend Jobs for Talend Big Data Platform V 6.3, HDP 2.7.3 is not list supported Hadoop distribution versions table.

Are you using a custom distribution in your tHDFSPut component?

Best regards

Sabrina