Error initializing SparkContext.

Highlighted

Error initializing SparkContext.

Hello community .
I am facing some issues while converting a map reduce code to spark. Please find the below error log and kindly help.
ting job FF_FF_SparkTest at 13:09 22/06/2016.
 
connecting to socket on port 3663
connected
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
SLF4J: Found binding in
SLF4J: See for an explanation.
SLF4J: Actual binding is of type
: org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
: org.apache.spark.SparkConf - In Spark 1.0 and later spark.local.dir will be overridden by the value set by the cluster manager (via SPARK_LOCAL_DIRS in mesos/standalone and LOCAL_DIRS in YARN).
: org.apache.hadoop.hdfs.DFSClient - DataStreamer Exception
java.nio.channels.UnresolvedAddressException
      at sun.nio.ch.Net.checkAddress(Unknown Source)
      at sun.nio.ch.SocketChannelImpl.connect(Unknown Source)
      at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192)
      at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
      at org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1577)
      at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1318)
      at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1271)
      at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:464)
: org.apache.spark.SparkContext - Error initializing SparkContext.
java.io.IOException: DataStreamer Exception:
      at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:578)
Caused by: java.nio.channels.UnresolvedAddressException
      at sun.nio.ch.Net.checkAddress(Unknown Source)
      at sun.nio.ch.SocketChannelImpl.connect(Unknown Source)
      at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192)
      at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
      at org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1577)
      at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1318)
      at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1271)
      at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:464)
java.io.IOException: DataStreamer Exception:
      at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:578)
Caused by: java.nio.channels.UnresolvedAddressException
      at sun.nio.ch.Net.checkAddress(Unknown Source)
      at sun.nio.ch.SocketChannelImpl.connect(Unknown Source)
      at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192)
      at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
      at org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1577)
      at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1318)
: org.apache.spark.SparkContext - Error stopping SparkContext after init error.
java.lang.NullPointerException
      at org.apache.spark.network.netty.NettyBlockTransferService.close(NettyBlockTransferService.scala:152)
      at org.apache.spark.storage.BlockManager.stop(BlockManager.scala:1216)
      at org.apache.spark.SparkEnv.stop(SparkEnv.scala:96)
      at org.apache.spark.SparkContext.stop(SparkContext.scala:1659)
      at org.apache.spark.SparkContext.<init>(SparkContext.scala:565)
      at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:61)
      at spark_test.ff_ff_sparktest_0_1.FF_FF_SparkTest.runJobInTOS(FF_FF_SparkTest.java:1222)
      at spark_test.ff_ff_sparktest_0_1.FF_FF_SparkTest.main(FF_FF_SparkTest.java:1117)
: spark_test.ff_ff_sparktest_0_1.FF_FF_SparkTest - TalendJob: 'FF_FF_SparkTest' - Failed with exit code: 1.
      at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1271)
      at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:464)
Job FF_FF_SparkTest ended at 13:13 22/06/2016.
Moderator

Re: Error initializing SparkContext.

Hi,
Could you please indicate on which build version you got this issue?
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.

Re: Error initializing SparkContext.

Hi,
Talend build version 6.1.1.20151214_1327.
Also I am attaching the latest errors that I am getting kindly help me with that.
Regards
Salil 
Talend_Spark_Errors.txt_20160627-0032.txt
Moderator

Re: Error initializing SparkContext.

Hi,
The attachment Talend_Spark_Errors.txt_20160627-0032.txt is not available on forum. Could you please take a look at it?
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star QBM
One Star

Re: Error initializing SparkContext.

Hello , I have exactly the same problem. 
Have you got  any explication or possibility. 
Please.
 Thank you
Quentin
One Star

Re: Error initializing SparkContext.

I presume this is spark batch job 
You need to double check your spark configuration inside the job.  
1- in the run tab.  Go to spark configuration.  
Make sure the connection is imported from the repository 
2-inside the hdfs connection component ( if you are using it) use the same connection from the repository 
Try with a minimum component to isolate the issue 
One Star QBM
One Star

Re: Error initializing SparkContext.

Hello, thank you for your answer 
I use the repository for everything. My cluster is correctly configured. I tried a simple transfer of file But it's still doesn't work.
Four Stars

Re: Error initializing SparkContext.

I am facing the same issue. I have tried all the options mentioned in the thread but nothing is working. I am reading json file as a source and my job is a spark big data job.

Cloud Free Trial

Try Talend Cloud free for 30 days.

Tutorial

Introduction to Talend Open Studio for Data Integration.

Definitive Guide to Data Integration

Practical steps to developing your data integration strategy.

Definitive Guide to Data Quality

Create systems and workflow to manage clean data ingestion and data transformation.