One Star

Connecting from Talend to Spark 1.6.0 Cluster

Hello,
I have created a job in talend to connect to a Spark 1.6.0 Cluster.
When running the job from Talend an error is thrown.
Configuration:
Spakr 1.6 , spark-mode : standalone, distrubition : custom
Any help on this would be highly appreciated since this issue is a real showstopper !!
java.lang.NoClassDefFoundError: org/apache/spark/scheduler/SparkListener
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
at java.lang.Class.getMethod0(Class.java:3018)
at java.lang.Class.getMethod(Class.java:1784)
at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.scheduler.SparkListener
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 7 more
Error: A JNI error has occurred, please check your installation and try again

Exception in thread "main"
2 REPLIES
Moderator

Re: Connecting from Talend to Spark 1.6.0 Cluster

Hi,
Could you please indicate the build version you are using? Are you using jdk 1.8? Have you ever executed your other jobs in your Talend studio successfully?
Screenshots of job setting will be preferred.
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: Connecting from Talend to Spark 1.6.0 Cluster

I think you need just to configure the job to use Spark.
Go to "Run" tab -> Spark configuration -> load the properties from the repository ( that i presume you created under metadata as hadoop cluster )