[Solved] tHiveRow for Hadoop HA

Anonymous — Sat, 16 Nov 2024 11:20:53 GMT

Hi all,
I am trying to load a file from HDFS into a Hive table using tHiveRow component.
Talend BigData Studio 5.6
Hadoop Hortonworks 2.1 with High Availability feature
Under advanced settings for tHiveRow I have:
dfs.client.use.datanode.hostname    true
dfs.datanode.use.datanode.hostname    true
dfs.nameservices    HDPCLUSTER
dfs.ha.namenodes.HDPCLUSTER    no,nt
dfs.namenode.rpc-address.HDPCLUSTER.no    hdp:8020
dfs.namenode.rpc-address.HDPCLUSTER.nt    hdp1:8020
dfs.client.failover.proxy.provider.HDPCLUSTER    org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
When I run the job it's throwin the following exception:
FAILED: RuntimeException java.io.IOException: Couldn't create proxy provider class org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
java.lang.RuntimeException: java.io.IOException: Couldn't create proxy provider class org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
        at org.apache.hadoop.hive.ql.Context.getScratchDir(Context.java:222)
        at org.apache.hadoop.hive.ql.Context.getExternalScratchDir(Context.java:278)
        at org.apache.hadoop.hive.ql.Context.getExternalTmpPath(Context.java:344)
        at org.apache.hadoop.hive.ql.parse.LoadSemanticAnalyzer.analyzeInternal(LoadSemanticAnalyzer.java:246)
        at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
        at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:426)
        at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:322)
        at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:976)
        at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:969)
        at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:99)
        at org.apache.hive.service.cli.operation.SQLOperation.run(SQLOperation.java:172)
        at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:231)
        at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:218)
        at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:233)
        at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:346)
        at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:230)
        at data_integration.hiveexample_0_1.HiveExample.tHiveRow_1Process(HiveExample.java:549)
        at data_integration.hiveexample_0_1.HiveExample.tLibraryLoad_1Process(HiveExample.java:383)
        at data_integration.hiveexample_0_1.HiveExample.runJobInTOS(HiveExample.java:801)
        at data_integration.hiveexample_0_1.HiveExample.main(HiveExample.java:682)
Caused by: java.io.IOException: Couldn't create proxy provider class org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
        at org.apache.hadoop.hdfs.NameNodeProxies.createFailoverProxyProvider(NameNodeProxies.java:456)
        at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:148)
        at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:579)
        at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:524)
        at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:146)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2397)
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:89)
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2431)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2413)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:368)
        at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
        at org.apache.hadoop.hive.ql.Context.getScratchDir(Context.java:210)
        ... 19 more
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at org.apache.hadoop.hdfs.NameNodeProxies.createFailoverProxyProvider(NameNodeProxies.java:445)
        ... 30 more
Caused by: java.lang.RuntimeException: Could not find any configured addresses for URI hdfs://HDPCLUSTER/tmp/hive-hive/hive_2015-01-16_12-53-36_655_5380462070013163379-1
        at org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider.<init>(ConfiguredFailoverProxyProvider.java:93)
        ... 35 more
Any ideas why?
Thank you

Re: [Solved] tHiveRow for Hadoop HA

Anonymous — Tue, 20 Jan 2015 09:17:47 GMT

Solved by adding a tHiveConnection component and use this existing connection in tHiveRow

topic [Solved] tHiveRow for Hadoop HA in Talend Studio

[Solved] tHiveRow for Hadoop HA

Re: [Solved] tHiveRow for Hadoop HA