[resolved] TOS BigData connect to hadoop cloudera cdh4.1.3

[Anonymous]

Dear all
We are using TBD 5.2.2.
Hadoop Cloudera CDH4.1.3 .
the defaulth version in TBD for CDH4 is CDH4.0.1,so we import jar of 4.1.3 into the component configuration file.
Maybe it works,but get errors:
xception in thread "main" java.lang.Error: java.lang.NoClassDefFoundError: org/apache/commons/cli/ParseException
at projectname111.datatransfer_0_1.DataTransfer.tMSSqlInput_1Process(DataTransfer.java:652)
at projectname111.datatransfer_0_1.DataTransfer.runJobInTOS(DataTransfer.java:835)
at projectname111.datatransfer_0_1.DataTransfer.main(DataTransfer.java:699)
Caused by: java.lang.NoClassDefFoundError: org/apache/commons/cli/ParseException
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:378)
disconnected
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:356)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:124)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2218)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:80)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2252)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2234)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:300)
at org.apache.hadoop.fs.FileSystem$1.run(FileSystem.java:146)
at org.apache.hadoop.fs.FileSystem$1.run(FileSystem.java:144)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Unknown Source)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:144)
at projectname111.datatransfer_0_1.DataTransfer.tMSSqlInput_1Process(DataTransfer.java:452)
... 2 more
Caused by: java.lang.ClassNotFoundException: org.apache.commons.cli.ParseException
at java.net.URLClassLoader$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
... 17 more
Can anybody give some regustion?
Thanks very much!

Joe

[Anonymous]

Hi,
Perhaps this forum can point you a light https://community.talend.com/t5/Data-Quality-Preparation-and/Talend-and-HortonWorks-2-0a/td-p/107573
Best regards
Sabrina

[Anonymous]

Hi
I have tried tHDFSOutput.
Deleted ComponentsCache,and got the same error.
I have imported all the cdh4.1.3 jar that this component needs to D:\Talend\DataIntegration\TOS_BD-r99271-V5.2.2\plugins\org.talend.designer.components.bigdata_5.2.2.r99271\components\tHDFSOutput.

[Anonymous]

Hi,
Perhaps something wrong with that. How about choosing the "Custom definition" option in the latest version Talend Open Studio for BigData(5.3.0 r101800).
tHDFSOutput.
Best regards
Sabrina

[Anonymous]

Hi,
Did you try to make the original component work against your CDH4.1.3 before modifying it?
In 5.2.2, even if the dependencies are CDH4.0.1, the components work with CDH4.1.2. I have not tested with CDH4.1.3, but since it's a minor version of CDH, I guess it should work as well.
Anyway, depending to your error above, commons-cli is missing in your classpath. But please try to use the original component, else, we can't help you.

[Anonymous]

but 5.2.2 does not have a "custom" option!

[Anonymous]

Hi,
The "custom " option is our new feature which is only available in latest version(5.3.0 r101800), so far.
See the related jira bug https://jira.talendforge.org/browse/TDI-25583.
Best regards
Sabrina

[Anonymous · Accepted solution]

Hi Sabrina
you are right,TBD5.3.0 can fix this issue. What's more,new version is downward compatible. I can import the old jobs and they still work,without any modification of BigData components.
Thanks very much!
Best Regards,
Joe

[Anonymous]

Hi Joe,
Thanks for your feedback. Feel free post your issue on forum.
Best regards
Sabrina