topic Re: [resolved] Error in Spark job run via YARN Client in Talend Studio

[resolved] Error in Spark job run via YARN Client

Anonymous — Sat, 16 Nov 2024 10:16:08 GMT

Hello, I've create a Spark job and it runs fine in local mode through Talend, but when I try to use YARN Client mode I get the following error. It isn't clear what property in Talend might be configured incorrectly. Spark applications run outside of Talend do not have this issue. Has anyone else run into this?
java.lang.ClassCastException: org.apache.spark.deploy.SparkHadoopUtil cannot be cast to org.apache.spark.deploy.yarn.YarnSparkHadoopUtil
at org.apache.spark.deploy.yarn.YarnSparkHadoopUtil$.get(YarnSparkHadoopUtil.scala:163)
at org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:257)
at org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:561)
at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:115)
at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:59)
at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:141)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:497)
at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:61)
at local_project.project_demo_0_1.project_demo.runJobInTOS(project_demo.java:1985)
at local_project.project_demo_0_1.project_demo.main(project_demo.java:1868)

Re: [resolved] Error in Spark job run via YARN Client

Anonymous — Thu, 10 Nov 2016 03:10:40 GMT

Hi,
Could you please indicate on which build version you got this issue? Screenshots of job setting will be helpful for us to address your issue.
Best regards
Sabrina

Re: [resolved] Error in Spark job run via YARN Client

Anonymous — Thu, 12 Jan 2017 03:42:07 GMT

I worked further with Talend support to add additional parameters to the spark configuration to get past this issue.
I had to set my HDP version with these configs:
spark.driver.extraJavaOptions
spark.yarn.am.extraJavaOptions
In addition, set these:
spark.hadoop.mapreduce.application.framework.path
spark.hadoop.mapreduce.application.classpath