[resolved] Error in Spark job run via YARN Client

[Anonymous]

Hello, I've create a Spark job and it runs fine in local mode through Talend, but when I try to use YARN Client mode I get the following error. It isn't clear what property in Talend might be configured incorrectly.  Spark applications run outside of Talend do not have this issue.  Has anyone else run into this?
java.lang.ClassCastException: org.apache.spark.deploy.SparkHadoopUtil cannot be cast to org.apache.spark.deploy.yarn.YarnSparkHadoopUtil
at org.apache.spark.deploy.yarn.YarnSparkHadoopUtil$.get(YarnSparkHadoopUtil.scala:163)
at org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:257)
at org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:561)
at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:115)
at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:59)
at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:141)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:497)
at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:61)
at local_project.project_demo_0_1.project_demo.runJobInTOS(project_demo.java:1985)
at local_project.project_demo_0_1.project_demo.main(project_demo.java:1868)

[Anonymous]

Hi,
Could you please indicate on which build version you got this issue? Screenshots of job setting will be helpful for us to address your issue.
Best regards
Sabrina

[Anonymous · Accepted solution]

I worked further with Talend support to add additional parameters to the spark configuration to get past this issue.
I had to set my HDP version with these configs:
spark.driver.extraJavaOptions
spark.yarn.am.extraJavaOptions
In addition, set these: 
spark.hadoop.mapreduce.application.framework.path
spark.hadoop.mapreduce.application.classpath