topic Big Data Batch Job Spark Configuration error in Talend Studio

Big Data Batch Job Spark Configuration error

Anonymous — Sat, 16 Nov 2024 10:02:50 GMT

I am trying to run a big data batch job on spark and it is giving an exception to add hdp version in spark-env.sh I am using horton works HDP 2.5 distribution and added that to spark-env.sh file but I am still getting the same error. PFA of my error.

Re: Big Data Batch Job Spark Configuration error

Anonymous — Thu, 23 Feb 2017 22:34:27 GMT

Make sure to specify the following arguments in the spark configuration when using YARN (using HDP)
spark.driver.extraJavaOptions=""
spark.yarn.am.extraJavaOptions=""
spark.hadoop.mapreduce.application.framework.path=""
spark.hadoop.mapreduce.application.classpath=""

Re: Big Data Batch Job Spark Configuration error

laxminarayan_reddiar — Thu, 23 Feb 2017 22:55:16 GMT

Thanks for the reply. Can you please explain me in detail what to add in that arguments in quotes. Like I have added this already in my job

Re: Big Data Batch Job Spark Configuration error

Anonymous — Fri, 24 Feb 2017 15:13:53 GMT

You can usually find these in your YARN config in Ambari:
Example below where 2.3.2.0-2950 is your HDP version
spark.driver.extraJavaOptions="-Dhdp.version=2.3.2.0-2950"
spark.yarn.am.extraJavaOptions="-Dhdp.version=2.3.2.0-2950"
spark.hadoop.mapreduce.application.framework.path="$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:$PWD/mr-framework/hadoop/share/hadoop/tools/lib/*:/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure"
spark.hadoop.mapreduce.application.classpath="/hdp/apps/2.3.2.0-2950/mapreduce/mapreduce.tar.gz#mr-framework"