Cannot use tHDFSconnection and tHBASEconnection in the same Job

TalendSolutionExpert — Fri, 09 Feb 2024 18:22:49 GMT

Symptom

Cannot use tHDFSconnection and tHBASEconnection in the same Job.

Diagnosis

java.io.IOException: java.lang.reflect.InvocationTargetException 
 at org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:383)
 at org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:360)
 at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:244)
 at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:187)
 at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:149)
 at talend_project.hbase_connection1_0_1.HBASE_connection1.tHBaseInput_2Process(HBASE_connection1.java:4134)
 at talend_project.hbase_connection1_0_1.HBASE_connection1.tHBaseConnection_1Process(HBASE_connection1.java:3378)
 at talend_project.hbase_connection1_0_1.HBASE_connection1.tHDFSConnection_1Process(HBASE_connection1.java:3022)
 at talend_project.hbase_connection1_0_1.HBASE_connection1.tMysqlConnection_1Process(HBASE_connection1.java:2693)
 at talend_project.hbase_connection1_0_1.HBASE_connection1.tFileInputDelimited_1Process(HBASE_connection1.java:2470)
 at talend_project.hbase_connection1_0_1.HBASE_connection1.runJobInTOS(HBASE_connection1.java:5422)
 at talend_project.hbase_connection1_0_1.HBASE_connection1.main(HBASE_connection1.java:4713)

Caused by: java.lang.reflect.InvocationTargetException
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
 at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
 at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
 at org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:381)
 ... 11 more

Caused by: java.util.ServiceConfigurationError: org.apache.hadoop.fs.FileSystem: Provider org.apache.hadoop.hdfs.DistributedFileSystem could not be instantiated
 at java.util.ServiceLoader.fail(ServiceLoader.java:232)
 at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
 at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
 at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
 at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
 at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2400)
 at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2411)
 at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2428)
 at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:88)
 at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2467)
 at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2449)
 at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:367)
 at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:166)
 at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:351)
 at org.apache.hadoop.fs.Path.getFileSystem(Path.java:287)
 at org.apache.hadoop.hbase.util.DynamicClassLoader.<init>(DynamicClassLoader.java:104)
 at org.apache.hadoop.hbase.protobuf.ProtobufUtil.<clinit>(ProtobufUtil.java:163)
 at org.apache.hadoop.hbase.ClusterId.parseFrom(ClusterId.java:64)
 at org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:69)
 at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:83)
 at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.retrieveClusterId(HConnectionManager.java:794)
 at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:627)
 ... 16 more

Caused by: java.lang.NoSuchMethodError: org.apache.hadoop.conf.Configuration.addDeprecations([Lorg/apache/hadoop/conf/Configuration$DeprecationDelta;)V
 at org.apache.hadoop.hdfs.HdfsConfiguration.addDeprecatedKeys(HdfsConfiguration.java:66)
 at org.apache.hadoop.hdfs.HdfsConfiguration.<clinit>(HdfsConfiguration.java:31)
 at org.apache.hadoop.hdfs.DistributedFileSystem.<clinit>(DistributedFileSystem.java:116)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
 at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
 at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
 at java.lang.Class.newInstance(Class.java:442)
 at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
 ... 35 more
Job HBASE_connection1 ended at 13:11 27/09/2017. [exit code=1]

Both components were causing a jar conflict.

Solution

Lower the distribution version of the tHDFSconnection, after which the Job will run fine without any conflicts.

article Cannot use tHDFSconnection and tHBASEconnection in the same Job in Official Support Articles

Cannot use tHDFSconnection and tHBASEconnection in the same Job

Symptom

Diagnosis

Solution