Error initializing SparkContext.

Hello community .
I am facing some issues while converting a map reduce code to spark. Please find the below error log and kindly help.
ting job FF_FF_SparkTest at 13:09 22/06/2016.
 
connecting to socket on port 3663
connected
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
SLF4J: Found binding in
SLF4J: See for an explanation.
SLF4J: Actual binding is of type
: org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
: org.apache.spark.SparkConf - In Spark 1.0 and later spark.local.dir will be overridden by the value set by the cluster manager (via SPARK_LOCAL_DIRS in mesos/standalone and LOCAL_DIRS in YARN).
: org.apache.hadoop.hdfs.DFSClient - DataStreamer Exception
java.nio.channels.UnresolvedAddressException
      at sun.nio.ch.Net.checkAddress(Unknown Source)
      at sun.nio.ch.SocketChannelImpl.connect(Unknown Source)
      at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192)
      at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
      at org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1577)
      at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1318)
      at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1271)
      at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:464)
: org.apache.spark.SparkContext - Error initializing SparkContext.
java.io.IOException: DataStreamer Exception:
      at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:578)
Caused by: java.nio.channels.UnresolvedAddressException
      at sun.nio.ch.Net.checkAddress(Unknown Source)
      at sun.nio.ch.SocketChannelImpl.connect(Unknown Source)
      at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192)
      at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
      at org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1577)
      at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1318)
      at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1271)
      at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:464)
java.io.IOException: DataStreamer Exception:
      at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:578)
Caused by: java.nio.channels.UnresolvedAddressException
      at sun.nio.ch.Net.checkAddress(Unknown Source)
      at sun.nio.ch.SocketChannelImpl.connect(Unknown Source)
      at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192)
      at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
      at org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1577)
      at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1318)
: org.apache.spark.SparkContext - Error stopping SparkContext after init error.
java.lang.NullPointerException
      at org.apache.spark.network.netty.NettyBlockTransferService.close(NettyBlockTransferService.scala:152)
      at org.apache.spark.storage.BlockManager.stop(BlockManager.scala:1216)
      at org.apache.spark.SparkEnv.stop(SparkEnv.scala:96)
      at org.apache.spark.SparkContext.stop(SparkContext.scala:1659)
      at org.apache.spark.SparkContext.<init>(SparkContext.scala:565)
      at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:61)
      at spark_test.ff_ff_sparktest_0_1.FF_FF_SparkTest.runJobInTOS(FF_FF_SparkTest.java:1222)
      at spark_test.ff_ff_sparktest_0_1.FF_FF_SparkTest.main(FF_FF_SparkTest.java:1117)
: spark_test.ff_ff_sparktest_0_1.FF_FF_SparkTest - TalendJob: 'FF_FF_SparkTest' - Failed with exit code: 1.
      at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1271)
      at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:464)
Job FF_FF_SparkTest ended at 13:13 22/06/2016.
7 REPLIES
Moderator

Re: Error initializing SparkContext.

Hi,
Could you please indicate on which build version you got this issue?
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.

Re: Error initializing SparkContext.

Hi,
Talend build version 6.1.1.20151214_1327.
Also I am attaching the latest errors that I am getting kindly help me with that.
Regards
Salil 
Talend_Spark_Errors.txt_20160627-0032.txt
Moderator

Re: Error initializing SparkContext.

Hi,
The attachment Talend_Spark_Errors.txt_20160627-0032.txt is not available on forum. Could you please take a look at it?
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star QBM
One Star

Re: Error initializing SparkContext.

Hello , I have exactly the same problem. 
Have you got  any explication or possibility. 
Please.
 Thank you
Quentin
One Star

Re: Error initializing SparkContext.

I presume this is spark batch job 
You need to double check your spark configuration inside the job.  
1- in the run tab.  Go to spark configuration.  
Make sure the connection is imported from the repository 
2-inside the hdfs connection component ( if you are using it) use the same connection from the repository 
Try with a minimum component to isolate the issue 
One Star QBM
One Star

Re: Error initializing SparkContext.

Hello, thank you for your answer 
I use the repository for everything. My cluster is correctly configured. I tried a simple transfer of file But it's still doesn't work.
Four Stars

Re: Error initializing SparkContext.

I am facing the same issue. I have tried all the options mentioned in the thread but nothing is working. I am reading json file as a source and my job is a spark big data job.