Spark Kinesis Streaming job - NoClassDefFoundError: org/apache/spark/Logging

Five Stars

Spark Kinesis Streaming job - NoClassDefFoundError: org/apache/spark/Logging

I designed a simple Kinesis stream read job and write to a tLogRow component with the HDFS Configuration also set up and the Hadoop Cluster metadata as well set up.

 

Talend Version: 6.3.1
Build id: 20161216_1026

JRE: jre1.8.0_91

 

Error:

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/Logging
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.spark.streaming.kinesis.KinesisUtils.createStream(KinesisUtils.scala)
at oodle_demo_project.tg_mda_kinesis_template_0_1.tg_mda_kinesis_template.tKinesisInput_1Process(tg_mda_kinesis_template.java:436)
at oodle_demo_project.tg_mda_kinesis_template_0_1.tg_mda_kinesis_template.run(tg_mda_kinesis_template.java:786)
at oodle_demo_project.tg_mda_kinesis_template_0_1.tg_mda_kinesis_template.runJobInTOS(tg_mda_kinesis_template.java:747)
at oodle_demo_project.tg_mda_kinesis_template_0_1.tg_mda_kinesis_template.main(tg_mda_kinesis_template.java:630)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.Logging
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

 

Screenshots:

 

Spark Configuration.png

HDFS Configuration.pngSparm Streaming Job Design.png

 

Moderator

Re: Spark Kinesis Streaming job - NoClassDefFoundError: org/apache/spark/Logging

Hello,

Here is an article https://stackoverflow.com/questions/40287289/java-lang-noclassdeffounderror-org-apache-spark-logging...

Hope it will help.

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.