SecurityException when tElasticSearchOutput and tExtractJsonFields components are both used in a Spark Job

Problem Description

 

A Talend 6.4.1 Spark Job, that uses tElasticSearchOutput and tExtractJsonFields components
against CDH 5.10, fails at execution time with the following exception stack:

Caused by: java.lang.SecurityException: class "org.apache.commons.logging.impl.Log4JLogger"'s signer information
does not match signer information of other classes in the same package
at java.lang.ClassLoader.checkCerts(ClassLoader.java:895)
at java.lang.ClassLoader.preDefineClass(ClassLoader.java:665)
at java.lang.ClassLoader.defineClass(ClassLoader.java:758)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method) 

 

Root Cause

The built Job contains three different versions of the commons-logging JAR file, and mixing classes from these different libraries is causing this issue. 

  • commons-logging-1.1.1.jar
  • commons-logging-1.1.3.jar
  • commons-logging-1.2.jar

 

Solution

Workarounds

  • Create two separate Jobs, one for each component.
  • Update the Elasticsearch Spark library: in your Spark Job, use the tLibraryLoad component to load a more recent version of the Elasticsearch Spark JAR file.

 

Patch

If the workarounds are not usable or feasible for any reason, a patch is available. Contact Talend Support for additional information and assistance.

Version history
Revision #:
9 of 9
Last update:
‎09-29-2018 12:14 AM
Updated by: