Multithread with Hadoop

One Star

Multithread with Hadoop

Hello,
we have the following error when executing a job containing subjobs with the multithread option on in Hadoop (multithread option off it works, but we need it to be multithreaded)
The job is sent to Hadoop and then executed in Hadoop via an Oozie deployment with the Talend Studio v6.0.1 which is part of Talend Big Data Platform.

 Log Type: stderr
 
  Log Upload Time: Wed Oct 14 19:21:31 +0200 2015
 
  Log Length: 3899
  SLF4J: Class path contains multiple SLF4J bindings.
  SLF4J: Found binding in [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  SLF4J: Found binding in [jar:file:/opt/yarn/nm/filecache/3692/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  SLF4J: See for an explanation.
  SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
  Exception in component tRunJob_1
  java.lang.RuntimeException: Child job returns 1. It doesn't terminate normally.
  Exception in thread "main" java.lang.UnsupportedClassVersionError: pbi/test_bd_batch03_0_1/test_BD_batch03 : Unsupported major.minor version 51.0
  at java.lang.ClassLoader.defineClass1(Native Method)
  at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
  at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
  at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
  at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
  at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
  at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
  at java.security.AccessController.doPrivileged(Native Method)
  at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
  at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
  Could not find the main class: pbi.test_bd_batch03_0_1.test_BD_batch03.  Program will exit.
 
  at pbi.test6_0_1.test6.tRunJob_1Process(test6.java:1051)
  at pbi.test6_0_1.test6.runJobInTOS(test6.java:1428)
  at pbi.test6_0_1.test6.main(test6.java:1159)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:606)
  at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:57)
  at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:46)
  at org.apache.oozie.action.hadoop.JavaMain.main(JavaMain.java:38)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:606)
  at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:228)
  at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
  at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
  at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:370)
  at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:295)
  at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:181)
  at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:224)
  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
  at java.util.concurrent.FutureTask.run(FutureTask.java:262)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  at java.lang.Thread.run(Thread.java:745)
  Intercepting System.exit(1)
  Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.JavaMain], exit code [1]
  log4j:WARN No appenders could be found for logger (org.apache.hadoop.mapreduce.v2.app.MRAppMaster).
  log4j:WARN Please initialize the log4j system properly.
  log4j:WARN See for more info.


Thank you for your help.

Cheers,
Nabil.

Ten Stars

Re: Multithread with Hadoop

This looks like it is caused by an incorrect Java version running on your Hadoop cluster. "Unsupported major.minor version 51.0" it looks you are running a version less than 7 of Java on your Hadoop cluster.

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Put Massive Amounts of Data to Work

Learn how to make your data more available, reduce costs and cut your build time

Watch Now

How OTTO Utilizes Big Data to Deliver Personalized Experiences

Read about OTTO's experiences with Big Data and Personalized Experiences

Blog

Talend Integration with Databricks

Take a look at this video about Talend Integration with Databricks

Watch Now