Running a MapReduce job using a Hortonworks cluster throws an illegal argument exception on mapreduce.application.framework.path

Talend Version          6.3.1

Summary

When running a MR Job using Studio, the Job fails with the following error:

http://XXXXX.XXXX.XXXX:XXXX/ws/v1/timeline/ 2017-03-06 00:59:52,005 INFO [main] client.RMProxy (RMProxy.java:createRMProxy(98)) - Connecting to ResourceManager at <Hostname>/<IP>:<PORT> Exception in thread "main" java.lang.IllegalArgumentException: Unable to parse '/hdp/apps/${hdp.version}/mapreduce/mapreduce.tar.gz#mr-framework' as a URI, check the setting for mapreduce.application.framework.path
Additional Versions  
Product Talend BigData, Talend Studio
Component  
Problem Description

Running an HA - HWX Cluster 2.5.3 with Talend 6.3.1, and while executing a MR Job, you get the following error:

http://XXXXX.XXXX.XXXX:XXXX/ws/v1/timeline/ 2017-03-06 00:59:52,005 INFO [main] client.RMProxy (RMProxy.java:createRMProxy(98)) - Connecting to ResourceManager at <Hostname>/<IP>:<PORT> Exception in thread "main" java.lang.IllegalArgumentException: Unable to parse '/hdp/apps/${hdp.version}/mapreduce/mapreduce.tar.gz#mr-framework' as a URI, check the setting for mapreduce.application.framework.path
org.apache.hadoop.mapreduce.JobSubmitter.addMRFrameworkToDistributedCache(JobSubmitter.java:443) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:142) at
org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290) at
org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:415) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at
org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
Problem root cause

According to Hortonworks, the issue happens because Ambari is unable to set the $hdp.version.

Even after setting the hdp version in Ambari, this doesn't fix the issue.

/hdp/apps/${hdp.version}/mapreduce/mapreduce.tar.gz#mr-framework
Solution

To resolve this issue, you need to pass the following parameter in the RUN view > Advanced Settings. Configure your memory appropriately, or your application will be killed by the AM container.

-Dhdp.version=2.3.6.0-3796
JIRA ticket number https://issues.apache.org/jira/browse/AMBARI-14751
Version history
Revision #:
3 of 3
Last update:
‎10-05-2017 11:39 AM
Updated by:
 
Contributors