Spark libjars can't be gotten

Eddie_Liu — Sat, 16 Nov 2024 00:06:43 GMT

we have following job, this job will submit to spark yarn. JOB_CLS_SAP_SALES_HEAD is a big data batch. when we deploy the job, we found only 2 Spark libjars were got, actually it need more libjars, it made NoClassDefFoundError. what's wrong? could anyone help on it?

[INFO ]:xxxxxxxxxxxxxxxx.job_db_sap_sales_order_scheduling_0_1.JOB_DB_SAP_SALES_ORDER_SCHEDULING - tRunJob_2 - The child job 'digital_factory_phase_i.job_stg_sap_vbap_0_1.JOB_STG_SAP_VBAP' is done.

[INFO ]:xxxxxxxxxxxxxxxx.job_db_sap_sales_order_scheduling_0_1.JOB_DB_SAP_SALES_ORDER_SCHEDULING - tRunJob_3 - The child job 'JOB_CLS_SAP_SALES_HEAD' starts on the version '0.1' with the context 'Default'.

[INFO ]:xxxxxxxxxxxxxxxx.job_db_sap_sales_order_scheduling_0_1.JOB_DB_SAP_SALES_ORDER_SCHEDULING - tRunJob_3 - The child job 'JOB_CLS_SAP_SALES_HEAD' is done.

[INFO ]:xxxxxxxxxxxxxxxx.job_cls_sap_sales_head_0_1.JOB_CLS_SAP_SALES_HEAD - TalendJob: 'JOB_CLS_SAP_SALES_HEAD' - Start.

[INFO ]:xxxxxxxxxxxxxxxx.job_cls_sap_sales_head_0_1.JOB_CLS_SAP_SALES_HEAD - Got 2 Spark libjars

[INFO ]:xxxxxxxxxxxxxxxx.job_cls_sap_sales_head_0_1.JOB_CLS_SAP_SALES_HEAD - Spark libjars : /data/talend7.2/Talend-7.2.1/jobserver/agent/TalendJobServersFiles/repository/DIGITAL_FACTORY_PHASE_I_JOB_DB_SAP_SALES_ORDER_SCHEDULING_20210526_161905_eCDws/JOB_DB_SAP_SALES_ORDER_SCHEDULING/../../../cache/lib/2427524776_551552229/routines.jar,../../../cache/lib/3704464275_3101559666/job_cls_sap_sales_head_0_1.jar

Caused by: java.lang.ClassNotFoundException: org.talend.bigdata.dataflow.hmap.PostProcessor

at java.net.URLClassLoader.findClass(URLClassLoader.java:381)

Re: Spark libjars can't be gotten

Anonymous — Fri, 28 May 2021 04:02:50 GMT

Hello,

We have replied to your another topic: https://community.talend.com/s/question/0D55b0000681fgmCAA/unix-shell-without-classpathreference-of-sub-jobs-java-lib

Could you please stick to one topic so that we can pay individual attention to your issue. Thanks for your understanding.

Best regards

Sabrina

topic Re: Spark libjars can't be gotten in Talend Studio

Spark libjars can't be gotten

Re: Spark libjars can't be gotten