rRunJob - Calling Standard Job from Spark Job Running in Yarn Cluster Mode ( MapR Cluser) is not working

Anonymous — Sat, 16 Nov 2024 03:37:41 GMT

Spark job in Running in yarn Cluster Mode in MapR Cluster.

On SubJobOK , at the end of job , I am invoking a Standard Job.

I am getting class not found Exception. Upon looking log , I fount that the jar is being referred from user cache. Path is something like this -

/tmp/hadoop-mapruat/nm-local-dir/usercache/useId/appcache/application_343434343434_1567/container_e8375037937593695474_1567_02_000001/subJobTest.jar

Exact same Job works fine if I run the Parent Spark Job in Yarn Client mode

Spark Job – Running in Yarn Cluster Mode – Calling Standard Job thru tRunJob - > Invocation of Sub Job in Failing

Spark Job – Running in Yarn Client Mode – Calling Standard Job thru tRunJob - > Invocation of Sub Job is working fine.

Question – Is the subjob invocation from Spark-Yarn-Mode supported in Talend ? if not what is the work around to invoke Standard job from Spark Job running in yarn Cluster Mode.

Re: rRunJob - Calling Standard Job from Spark Job Running in Yarn Cluster Mode ( MapR Cluser) is not working

Anonymous — Thu, 16 Jan 2020 18:55:06 GMT

we have decided not to invest in talend studio for our big data pipeline partly because its diffucult to get help on talend issues.

topic Re: rRunJob - Calling Standard Job from Spark Job Running in Yarn Cluster Mode ( MapR Cluser) is not working in Talend Studio

rRunJob - Calling Standard Job from Spark Job Running in Yarn Cluster Mode ( MapR Cluser) is not working

Re: rRunJob - Calling Standard Job from Spark Job Running in Yarn Cluster Mode ( MapR Cluser) is not working