Unix shell without classpath/reference of sub jobs' java lib

Eddie_Liu — Sat, 16 Nov 2024 00:07:30 GMT

We have a job that include 6 sub jobs, each sub job works well and fully tested, but when we try to run the main job, it throws following exceptions. The root cause is that the shell doesn't include sub jobs' jar reference

Anyone knows about it, how to add sub jobs' reference jar to shell.?

Exception in thread "main" java.lang.NoClassDefFoundError: org/talend/bigdata/dataflow/hmap/PostProcessor

at java.lang.Class.getDeclaredMethods0(Native Method)

at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)

at java.lang.Class.privateGetMethodRecursive(Class.java:3048)

at java.lang.Class.getMethod0(Class.java:3018)

at java.lang.Class.getMethod(Class.java:1784)

at org.apache.spark.deploy.yarn.ApplicationMaster.startUserApplication(ApplicationMaster.scala:679)

at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:461)

at org.apache.spark.deploy.yarn.ApplicationMaster.org$apache$spark$deploy$yarn$ApplicationMaster$$runImpl(ApplicationMaster.scala:305)

at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$1.apply$mcV$sp(ApplicationMaster.scala:245)

at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$1.apply(ApplicationMaster.scala:245)

at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:783)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:422)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)

at org.apache.spark.deploy.yarn.ApplicationMaster.doAsUser(ApplicationMaster.scala:782)

at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:244)

at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:807)

at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)

Caused by: java.lang.ClassNotFoundException: org.talend.bigdata.dataflow.hmap.PostProcessor

at java.net.URLClassLoader.findClass(URLClassLoader.java:381)

at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)

at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

Re: Unix shell without classpath/reference of sub jobs' java lib

Anonymous — Wed, 26 May 2021 07:51:07 GMT

Hello,

We are supposing that some of the JARs used by your components cannot be found in the current Java version.

What's JDK version are you using?

Best regards

Sabrina

Re: Unix shell without classpath/reference of sub jobs' java lib

Eddie_Liu — Thu, 27 May 2021 00:22:55 GMT

We are using Talend Data Fabric with default JDK. The root cause is, Talend created Unix Shell without the reference of sub job's jar. Do you know how to solve it? Thanks.

Re: Unix shell without classpath/reference of sub jobs' java lib

Anonymous — Thu, 27 May 2021 08:55:23 GMT

Hello,

With your subscription solution, please create a support case on talend support portal so that we could give you a remote assistance to deliver a patch if needed through support cycle with priority.

Best regards

Sabrina

topic Unix shell without classpath/reference of sub jobs' java lib in Talend Studio

Unix shell without classpath/reference of sub jobs' java lib

Re: Unix shell without classpath/reference of sub jobs' java lib

Re: Unix shell without classpath/reference of sub jobs' java lib

Re: Unix shell without classpath/reference of sub jobs' java lib