Multithread with Hadoop

Anonymous — Sat, 16 Nov 2024 10:59:20 GMT

Hello,
we have the following error when executing a job containing subjobs with the multithread option on in Hadoop (multithread option off it works, but we need it to be multithreaded)
The job is sent to Hadoop and then executed in Hadoop via an Oozie deployment with the Talend Studio v6.0.1 which is part of Talend Big Data Platform.

 Log Type: stderr
  
  Log Upload Time: Wed Oct 14 19:21:31 +0200 2015
  
  Log Length: 3899
  SLF4J: Class path contains multiple SLF4J bindings.
  SLF4J: Found binding in [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  SLF4J: Found binding in [jar:file:/opt/yarn/nm/filecache/3692/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  SLF4J: See  for an explanation.
  SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
  Exception in component tRunJob_1
  java.lang.RuntimeException: Child job returns 1. It doesn't terminate normally.
  Exception in thread "main" java.lang.UnsupportedClassVersionError: pbi/test_bd_batch03_0_1/test_BD_batch03 : Unsupported major.minor version 51.0
  at java.lang.ClassLoader.defineClass1(Native Method)
  at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
  at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
  at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
  at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
  at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
  at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
  at java.security.AccessController.doPrivileged(Native Method)
  at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
  at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
  Could not find the main class: pbi.test_bd_batch03_0_1.test_BD_batch03.  Program will exit.
  
  at pbi.test6_0_1.test6.tRunJob_1Process(test6.java:1051)
  at pbi.test6_0_1.test6.runJobInTOS(test6.java:1428)
  at pbi.test6_0_1.test6.main(test6.java:1159)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:606)
  at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:57)
  at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:46)
  at org.apache.oozie.action.hadoop.JavaMain.main(JavaMain.java:38)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:606)
  at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:228)
  at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
  at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
  at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:370)
  at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:295)
  at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:181)
  at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:224)
  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
  at java.util.concurrent.FutureTask.run(FutureTask.java:262)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  at java.lang.Thread.run(Thread.java:745)
  Intercepting System.exit(1)
  Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.JavaMain], exit code [1]
  log4j:WARN No appenders could be found for logger (org.apache.hadoop.mapreduce.v2.app.MRAppMaster).
  log4j:WARN Please initialize the log4j system properly.
  log4j:WARN See  for more info.

Thank you for your help.

Cheers,
Nabil.

Re: Multithread with Hadoop

Anonymous — Thu, 15 Oct 2015 11:05:59 GMT

This looks like it is caused by an incorrect Java version running on your Hadoop cluster. " Unsupported major.minor version 51.0" it looks you are running a version less than 7 of Java on your Hadoop cluster.

topic Multithread with Hadoop in Talend Studio

Multithread with Hadoop

Re: Multithread with Hadoop