Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.
Hi,
I'm using Talend real time big data edition.
I'm having an error when combining HDFSoutput components with Impala in the same job ( see scenario attached )
If we split reading from Impala and writing into HDFS into different jobs , it works fine
I do have a workaround but it would be great if we can use both in the same job.
Here is the error :
It looks like Impala loading a library or plugin that conflicts with HDFS libs
FYI : Talend version is 6.1.1
This is a standard ( not spark or mapreduce ) job
I'm using Talend real time big data edition.
I'm having an error when combining HDFSoutput components with Impala in the same job ( see scenario attached )
If we split reading from Impala and writing into HDFS into different jobs , it works fine
I do have a workaround but it would be great if we can use both in the same job.
Here is the error :
Exception in thread "main" java.lang.NoClassDefFoundError: org/htrace/Trace
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:214)
at com.sun.proxy.$Proxy7.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy8.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2095)
at org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1214)
disconnected
disconnected
disconnected
disconnected
at org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1210)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1210)
at XX.bv_accountcoms_0_1.BV_AccountComs.tHDFSInput_3Process(BV_AccountComs.java:5696)
It looks like Impala loading a library or plugin that conflicts with HDFS libs
FYI : Talend version is 6.1.1
This is a standard ( not spark or mapreduce ) job