Big Data Batch Job Spark Configuration error

[Anonymous]

I am trying to run a big data batch job on spark and it is giving an exception to add hdp version in spark-env.sh I am using horton works HDP 2.5 distribution and added that to spark-env.sh file but I am still getting the same error. PFA of my error.

[Anonymous]

Make sure to specify the following arguments in the spark configuration when using YARN (using HDP)
spark.driver.extraJavaOptions=""
spark.yarn.am.extraJavaOptions=""
spark.hadoop.mapreduce.application.framework.path=""
spark.hadoop.mapreduce.application.classpath=""

[laxminarayan_reddiar]

Thanks for the reply. Can you please explain me in detail what to add in that arguments in quotes. Like I have added this already in my job

[Anonymous]

You can usually find these in your YARN config in Ambari:
Example below where 2.3.2.0-2950 is your HDP version
spark.driver.extraJavaOptions="-Dhdp.version=2.3.2.0-2950"
spark.yarn.am.extraJavaOptions="-Dhdp.version=2.3.2.0-2950"
spark.hadoop.mapreduce.application.framework.path="$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:$PWD/mr-framework/hadoop/share/hadoop/tools/lib/*:/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure"
spark.hadoop.mapreduce.application.classpath="/hdp/apps/2.3.2.0-2950/mapreduce/mapreduce.tar.gz#mr-framework"