Issue with Talend Spark YARN Client Configuration for EMR
Hi All,
I have created a Talend Bigdata job using Spark YARN mode framework pointing to the EMR cluster (v5.15).
Please find the screenshot attached.
I am getting the following error and unable to solve the issue. I even added spark-yarn to the classpath, but no luck.
Can you please help me resolve the issue?
SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/D:/Talend_Studio/7_2_1/studio/configuration/.m2/repository/org/talend/libraries/slf4j-log4j12-1.7.10/6.0.0/slf4j-log4j12-1.7.10-6.0.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/D:/Talend_Studio/7_2_1/studio/configuration/.m2/repository/org/slf4j/slf4j-log4j12/1.7.16/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] [WARN ]: org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable [WARN ]: org.apache.spark.SparkConf - Note that spark.local.dir will be overridden by the value set by the cluster manager (via SPARK_LOCAL_DIRS in mesos/standalone/kubernetes and LOCAL_DIRS in YARN). [ERROR]: org.apache.spark.SparkContext - Error initializing SparkContext. org.apache.spark.SparkException: Could not parse Master URL: 'yarn' at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2784) at org.apache.spark.SparkContext.<init>(SparkContext.scala:493) at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58) at bdp_talend.sparkparquetjob_0_1.SparkParquetJob.runJobInTOS(SparkParquetJob.java:1143) at bdp_talend.sparkparquetjob_0_1.SparkParquetJob.main(SparkParquetJob.java:1039) [WARN ]: org.apache.spark.metrics.MetricsSystem - Stopping a MetricsSystem that is not running org.apache.spark.SparkException: Could not parse Master URL: 'yarn' at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2784) at org.apache.spark.SparkContext.<init>(SparkContext.scala:493) at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58) at bdp_talend.sparkparquetjob_0_1.SparkParquetJob.runJobInTOS(SparkParquetJob.java:1143) at bdp_talend.sparkparquetjob_0_1.SparkParquetJob.main(SparkParquetJob.java:1039)
