I am working with a cluster where we have custom hadoop 2.4. I am trying to use talend with spark components. For the Spark Connection components, I have the set the relevant SparkHost, SparkHome.
For the distribution, the two available options are Cloudera and Custom( unsupported). When the Custom( unsupported ) distribution is selected, there is a provision to choose the custom Hadoop version to include the relavant libraries. The options available here are: Cloudera, HortonWorks, MapR, Apache, Amazon EMR, PivotalHD. However for me, when I choose Cloudera it comes with Hadoop 2.3 and I am assuming that all essential libraries are missing, and hence I get an "NoClassDefFoundError" which leads to not being able to load a file in Spark via this Spark connection. Btw, the spark version I have is 1.0.0
I would like to know how to fix this and a way to get this version of Spark running with Hadoop Certification.
The error is copied and pasted below:
[statistics] connecting to socket on port 3637 [statistics] connected Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/api/java/JavaSparkContext at sparktest.sparktest_0_1.sparktest.tSparkConnection_2Process(sparktest.java:491) at sparktest.sparktest_0_1.sparktest.runJobInTOS(sparktest.java:1643) at sparktest.sparktest_0_1.sparktest.main(sparktest.java:1502) Caused by: java.lang.ClassNotFoundException: org.apache.spark.api.java.JavaSparkContext at java.net.URLClassLoader$1.run(URLClassLoader.java:372) at java.net.URLClassLoader$1.run(URLClassLoader.java:361) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:360) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 3 more [statistics] disconnected Job sparktest ended at 13:19 21/10/2014. [exit code=1]
Could you please indicate on which talend build version you got this issue? Here exists a jira issue:https://jira.talendforge.org/browse/TBD-3774 about "spark job can't work with HDP2.3".
Watch the recorded webinar!
Introduction to Talend Open Studio for Data Integration.
Test drive Talend's enterprise products.
Practical steps to developing your data integration strategy.