A Spark Job initiated from TAC fails with the error ‘java.lang.NoClassDefFoundError: org/apache/spark/api/java/function/PairFlatMapFunction’

Problem Description

While running a Spark Job on a Studio machine (localhost), the Job fails with the following error:

java.lang.NoClassDefFoundError: org/apache/spark/api/java/function/PairFlatMapFunction

To fix this issue, the park-core*.jar file was updated in the Studio m2 location, C:\Talend\6.4.1\studio\configuration\.m2\repository\org\talend\libraries, then Job runs successfully.

 

The m2 location of Studio is replicated in CommandLine. The Job is regenerated successfully from TAC, but the Job fails with the following error:

java.lang.NoClassDefFoundError: org/apache/spark/api/java/function/PairFlatMapFunction

 

Root Cause

Using the -verbose:class jvm option and running the Job from TAC, shows that the old version of JAR files are loaded from the Job server's cache, C:\Talend\6.4.1\jobserver\agent\TalendJobServersFiles\cache\lib.

 

Solution

To fix the issue, clear the old cached files from the C:\Talend\6.4.1\jobserver\agent\TalendJobServersFiles\cache\lib directory.

Version history
Revision #:
7 of 7
Last update:
‎02-24-2019 11:12 PM
Updated by: