Cannot use tHDFSconnection and tHBASEconnection in the same Job

[TalendSolutionExpert]

Symptom

Cannot use tHDFSconnection and tHBASEconnection in the same Job.

Diagnosis

java.io.IOException: java.lang.reflect.InvocationTargetException 
 at org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:383)
 at org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:360)
 at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:244)
 at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:187)
 at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:149)
 at talend_project.hbase_connection1_0_1.HBASE_connection1.tHBaseInput_2Process(HBASE_connection1.java:4134)
 at talend_project.hbase_connection1_0_1.HBASE_connection1.tHBaseConnection_1Process(HBASE_connection1.java:3378)
 at talend_project.hbase_connection1_0_1.HBASE_connection1.tHDFSConnection_1Process(HBASE_connection1.java:3022)
 at talend_project.hbase_connection1_0_1.HBASE_connection1.tMysqlConnection_1Process(HBASE_connection1.java:2693)
 at talend_project.hbase_connection1_0_1.HBASE_connection1.tFileInputDelimited_1Process(HBASE_connection1.java:2470)
 at talend_project.hbase_connection1_0_1.HBASE_connection1.runJobInTOS(HBASE_connection1.java:5422)
 at talend_project.hbase_connection1_0_1.HBASE_connection1.main(HBASE_connection1.java:4713)

Caused by: java.lang.reflect.InvocationTargetException
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
 at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
 at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
 at org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:381)
 ... 11 more

Caused by: java.util.ServiceConfigurationError: org.apache.hadoop.fs.FileSystem: Provider org.apache.hadoop.hdfs.DistributedFileSystem could not be instantiated
 at java.util.ServiceLoader.fail(ServiceLoader.java:232)
 at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
 at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
 at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
 at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
 at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2400)
 at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2411)
 at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2428)
 at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:88)
 at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2467)
 at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2449)
 at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:367)
 at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:166)
 at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:351)
 at org.apache.hadoop.fs.Path.getFileSystem(Path.java:287)
 at org.apache.hadoop.hbase.util.DynamicClassLoader.<init>(DynamicClassLoader.java:104)
 at org.apache.hadoop.hbase.protobuf.ProtobufUtil.<clinit>(ProtobufUtil.java:163)
 at org.apache.hadoop.hbase.ClusterId.parseFrom(ClusterId.java:64)
 at org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:69)
 at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:83)
 at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.retrieveClusterId(HConnectionManager.java:794)
 at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:627)
 ... 16 more

Caused by: java.lang.NoSuchMethodError: org.apache.hadoop.conf.Configuration.addDeprecations([Lorg/apache/hadoop/conf/Configuration$DeprecationDelta;)V
 at org.apache.hadoop.hdfs.HdfsConfiguration.addDeprecatedKeys(HdfsConfiguration.java:66)
 at org.apache.hadoop.hdfs.HdfsConfiguration.<clinit>(HdfsConfiguration.java:31)
 at org.apache.hadoop.hdfs.DistributedFileSystem.<clinit>(DistributedFileSystem.java:116)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
 at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
 at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
 at java.lang.Class.newInstance(Class.java:442)
 at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
 ... 35 more
Job HBASE_connection1 ended at 13:11 27/09/2017. [exit code=1]

Both components were causing a jar conflict.

Solution

Lower the distribution version of the tHDFSconnection, after which the Job will run fine without any conflicts.