Error initializing SparkContext.

Anonymous — Sat, 16 Nov 2024 10:33:42 GMT

Hello community .
I am facing some issues while converting a map reduce code to spark. Please find the below error log and kindly help.
ting job FF_FF_SparkTest at 13:09 22/06/2016.

connecting to socket on port 3663
connected
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
SLF4J: Found binding in
SLF4J: See for an explanation.
SLF4J: Actual binding is of type
: org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
: org.apache.spark.SparkConf - In Spark 1.0 and later spark.local.dir will be overridden by the value set by the cluster manager (via SPARK_LOCAL_DIRS in mesos/standalone and LOCAL_DIRS in YARN).
: org.apache.hadoop.hdfs.DFSClient - DataStreamer Exception
java.nio.channels.UnresolvedAddressException
at sun.nio.ch.Net.checkAddress(Unknown Source)
at sun.nio.ch.SocketChannelImpl.connect(Unknown Source)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1577)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1318)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1271)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:464)
: org.apache.spark.SparkContext - Error initializing SparkContext.
java.io.IOException: DataStreamer Exception:
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:578)
Caused by: java.nio.channels.UnresolvedAddressException
at sun.nio.ch.Net.checkAddress(Unknown Source)
at sun.nio.ch.SocketChannelImpl.connect(Unknown Source)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1577)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1318)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1271)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:464)
java.io.IOException: DataStreamer Exception:
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:578)
Caused by: java.nio.channels.UnresolvedAddressException
at sun.nio.ch.Net.checkAddress(Unknown Source)
at sun.nio.ch.SocketChannelImpl.connect(Unknown Source)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1577)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1318)
: org.apache.spark.SparkContext - Error stopping SparkContext after init error.
java.lang.NullPointerException
at org.apache.spark.network.netty.NettyBlockTransferService.close(NettyBlockTransferService.scala:152)
at org.apache.spark.storage.BlockManager.stop(BlockManager.scala:1216)
at org.apache.spark.SparkEnv.stop(SparkEnv.scala:96)
at org.apache.spark.SparkContext.stop(SparkContext.scala:1659)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:565)
at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:61)
at spark_test.ff_ff_sparktest_0_1.FF_FF_SparkTest.runJobInTOS(FF_FF_SparkTest.java:1222)
at spark_test.ff_ff_sparktest_0_1.FF_FF_SparkTest.main(FF_FF_SparkTest.java:1117)
: spark_test.ff_ff_sparktest_0_1.FF_FF_SparkTest - TalendJob: 'FF_FF_SparkTest' - Failed with exit code: 1.
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1271)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:464)
Job FF_FF_SparkTest ended at 13:13 22/06/2016.

Re: Error initializing SparkContext.

Anonymous — Mon, 27 Jun 2016 07:57:44 GMT

Hi,
Could you please indicate on which build version you got this issue?
Best regards
Sabrina

Re: Error initializing SparkContext.

Anonymous — Mon, 27 Jun 2016 08:32:39 GMT

Hi,
Talend build version 6.1.1.20151214_1327.
Also I am attaching the latest errors that I am getting kindly help me with that.
Regards
Salil
Talend_Spark_Errors.txt_20160627-0032.txt

Re: Error initializing SparkContext.

Anonymous — Tue, 28 Jun 2016 10:37:36 GMT

Hi,
The attachment Talend_Spark_Errors.txt_20160627-0032.txt is not available on forum. Could you please take a look at it?
Best regards
Sabrina

Re: Error initializing SparkContext.

Anonymous — Mon, 11 Jul 2016 10:09:49 GMT

Hello , I have exactly the same problem.
Have you got any explication or possibility.
Please.
Thank you
Quentin

Re: Error initializing SparkContext.

Anonymous — Tue, 26 Jul 2016 21:49:00 GMT

I presume this is spark batch job
You need to double check your spark configuration inside the job.
1- in the run tab. Go to spark configuration.
Make sure the connection is imported from the repository
2-inside the hdfs connection component ( if you are using it) use the same connection from the repository
Try with a minimum component to isolate the issue

Re: Error initializing SparkContext.

Anonymous — Wed, 10 Aug 2016 08:46:59 GMT

Hello, thank you for your answer
I use the repository for everything. My cluster is correctly configured. I tried a simple transfer of file But it's still doesn't work.

Re: Error initializing SparkContext.

Anonymous — Tue, 05 Jun 2018 11:56:39 GMT

I am facing the same issue. I have tried all the options mentioned in the thread but nothing is working. I am reading json file as a source and my job is a spark big data job.

topic Re: Error initializing SparkContext. in Talend Studio

Error initializing SparkContext.

Re: Error initializing SparkContext.

Re: Error initializing SparkContext.

Re: Error initializing SparkContext.

Re: Error initializing SparkContext.

Re: Error initializing SparkContext.

Re: Error initializing SparkContext.

Re: Error initializing SparkContext.