Problem when launching a batch mode job

One Star

Problem when launching a batch mode job

Hello,
I am using Talend Fabric 6.2.1 and hortonworks 2.4.
I am trying to launch the demo job of big data batch Job MR_count_code, I changed the configuration to adapt it to my virtual machine which installed hortonworks,
When i launched a standard job, it worked, but in a big data batch job always have this error
connecting to socket on port 3774
connected
: org.apache.hadoop.mapreduce.JobResourceUploader - No job jar file set.  User classes may not be found. See Job or Job#setJar(String).
Running job: job_1478076384405_0018
 map 0% reduce 0%
Job complete: job_1478076384405_0018
Counters: 0
Job Failed: Application application_1478076384405_0018 failed 2 times due to AM Container for appattempt_1478076384405_0018_000002 exited with  exitCode: 255
For more detailed output, check application tracking page:, click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_e02_1478076384405_0018_02_000001
Exit code: 255
Stack trace: ExitCodeException exitCode=255:
    at org.apache.hadoop.util.Shell.runCommand(Shell.java:576)
    at org.apache.hadoop.util.Shell.run(Shell.java:487)
    at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:753)
    at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:303)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

Container exited with a non-zero exit code 255
Failing this attempt. Failing the application.
java.io.IOException: Job failed!
    at org.talend.hadoop.mapred.lib.MRJobClient.runJob(MRJobClient.java:166)
    at demo.mr_count_code_0_1.MR_Count_Code.runMRJob(MR_Count_Code.java:2804)
    at demo.mr_count_code_0_1.MR_Count_Code.access$1(MR_Count_Code.java:2794)
    at demo.mr_count_code_0_1.MR_Count_Code$1.run(MR_Count_Code.java:2615)
    at demo.mr_count_code_0_1.MR_Count_Code$1.run(MR_Count_Code.java:1)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
    at demo.mr_count_code_0_1.MR_Count_Code.tHDFSInput_1Process(MR_Count_Code.java:2530)
    at demo.mr_count_code_0_1.MR_Count_Code.run(MR_Count_Code.java:2772)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
    at demo.mr_count_code_0_1.MR_Count_Code.runJobInTOS(MR_Count_Code.java:2711)
    at demo.mr_count_code_0_1.MR_Count_Code.main(MR_Count_Code.java:2690)
disconnected
Job MR_Count_Code terminé à 13:19 02/11/2016.
Employee

Re: Problem when launching a batch mode job

Did you check the container logs on your hadoop cluster ? Also change your log settings to debug and check for more errors.
One Star

Re: Problem when launching a batch mode job

Thanks you, for your respond,
2016-11-02 17:42:20,184 FATAL org.apache.hadoop.yarn.event.AsyncDispatcher: Error in dispatcher thread
java.lang.IllegalArgumentException: Unable to parse '/hdp/apps/${hdp.version}/mapreduce/mapreduce.tar.gz#mr-framework' as a URI, check the setting for mapreduce.application.framework.path
    at org.apache.hadoop.mapreduce.v2.util.MRApps.getMRFrameworkName(MRApps.java:181)
    at org.apache.hadoop.mapreduce.v2.util.MRApps.setMRFrameworkClasspath(MRApps.java:206)
    at org.apache.hadoop.mapreduce.v2.util.MRApps.setClasspath(MRApps.java:258)
    at org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.getInitialClasspath(TaskAttemptImpl.java:621)
    at org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.createCommonContainerLaunchContext(TaskAttemptImpl.java:757)
    at org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.createContainerLaunchContext(TaskAttemptImpl.java:821)
    at org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl$ContainerAssignedTransition.transition(TaskAttemptImpl.java:1557)
    at org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl$ContainerAssignedTransition.transition(TaskAttemptImpl.java:1534)
    at org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:362)
    at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
    at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
    at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
    at org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:1084)
    at org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:145)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:1368)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:1360)
    at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:183)
    at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:109)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.URISyntaxException: Illegal character in path at index 11: /hdp/apps/${hdp.version}/mapreduce/mapreduce.tar.gz#mr-framework
    at java.net.URI$Parser.fail(URI.java:2848)
    at java.net.URI$Parser.checkChars(URI.java:3021)
    at java.net.URI$Parser.parseHierarchical(URI.java:3105)
    at java.net.URI$Parser.parse(URI.java:3063)
    at java.net.URI.<init>(URI.java:588)
    at org.apache.hadoop.mapreduce.v2.util.MRApps.getMRFrameworkName(MRApps.java:179)
    ... 18 more

Best Regards.
Six Stars

Re: Problem when launching a batch mode job

Is that get fixed ? I am also facing the same issue and want to know how to fix this .
Employee

Re: Problem when launching a batch mode job

This is an issue with Hortonworks. The $hdp.version is not resolved by most MR clients. 
Set the mapreduce.application.classpath explcitly inside talend job and dont use the $hdp.version, use the actual value. 
Six Stars

Re: Problem when launching a batch mode job

could you please advise on this. I am not able to find or locate where to explicitly set the value.
I have checked the code and I am not able to find the ${hdp.version} variable  and path setting for  mapreduce.application.classpath .
Six Stars

Re: Problem when launching a batch mode job

Hi Amula, yes, i was able to set up the value in hadopp configruation /hadoop property. Now version issue is resolved , but i am facing other issue.
Diagnostics: Exception from container-launch.
Container id: container_e38_1481903964317_0019_02_000001
Exit code: 1
Stack trace: org.apache.hadoop.yarn.server.nodemanager.containermanager.runtime.ContainerExecutionException: Launch container failed
      at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DefaultLinuxContainerRuntime.launchContainer(DefaultLinuxContainerRuntime.java:109)
      at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DelegatingLinuxContainerRuntime.launchContainer(DelegatingLinuxContainerRuntime.java:89)
      at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:392)
      at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:317)
      at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:83)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:745)
 
Shell output: main : command provided 1
main : run as user is xxxxx
main : requested yarn user is xxxxx
Getting exit code file...
Creating script paths...
Writing pid file...
Writing to tmp file /data/c/hadoop/yarn/local/nmPrivate/application_1481903964317_0019/container_e38_1481903964317_0019_02_000001/container_e38_1481903964317_0019_02_000001.pid.tmp
Writing to cgroup task files...
Creating local dirs...
Launching container...
Getting exit code file...
Creating script paths... 
Container exited with a non-zero exit code 1
Failing this attempt. Failing the application.