Issue with Talend Spark YARN Client Configuration for EMR

Anonymous — Sat, 16 Nov 2024 02:07:44 GMT

Hi All,

I have created a Talend Bigdata job using Spark YARN mode framework pointing to the EMR cluster (v5.15).

Please find the screenshot attached.

I am getting the following error and unable to solve the issue. I even added spark-yarn to the classpath, but no luck.

Can you please help me resolve the issue?

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/D:/Talend_Studio/7_2_1/studio/configuration/.m2/repository/org/talend/libraries/slf4j-log4j12-1.7.10/6.0.0/slf4j-log4j12-1.7.10-6.0.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/D:/Talend_Studio/7_2_1/studio/configuration/.m2/repository/org/slf4j/slf4j-log4j12/1.7.16/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
[WARN ]: org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[WARN ]: org.apache.spark.SparkConf - Note that spark.local.dir will be overridden by the value set by the cluster manager (via SPARK_LOCAL_DIRS in mesos/standalone/kubernetes and LOCAL_DIRS in YARN).
[ERROR]: org.apache.spark.SparkContext - Error initializing SparkContext.
org.apache.spark.SparkException: Could not parse Master URL: 'yarn'
at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2784)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:493)
at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
at bdp_talend.sparkparquetjob_0_1.SparkParquetJob.runJobInTOS(SparkParquetJob.java:1143)
at bdp_talend.sparkparquetjob_0_1.SparkParquetJob.main(SparkParquetJob.java:1039)
[WARN ]: org.apache.spark.metrics.MetricsSystem - Stopping a MetricsSystem that is not running
org.apache.spark.SparkException: Could not parse Master URL: 'yarn'
at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2784)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:493)
at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
at bdp_talend.sparkparquetjob_0_1.SparkParquetJob.runJobInTOS(SparkParquetJob.java:1143)
at bdp_talend.sparkparquetjob_0_1.SparkParquetJob.main(SparkParquetJob.java:1039)

Thanks,

Lakshmi

topic Issue with Talend Spark YARN Client Configuration for EMR in Talend Studio

Issue with Talend Spark YARN Client Configuration for EMR