Error when running a MapReduce Job with MapR 4.0.2

Overview

When using Talend Big Data and Talend Studio v. 6.1.1, an error results when running a MapReduce Job with a MapR 4.0.2 cluster using a MapR 4.0.2 client.

 

Symptoms/description

You are attempting to run a MapReduce Job against a 4.0.2 MapR cluster using a MapR 4.0.2 client. Studio is using the 4.0.1 libraries. You get an error message that the job failed to initialize the cluster and your mapreduce.framework.name is not set up.

 

If you try with the 4.1.0 libraries, you get the following error:

ERROR JniCommon fs/client/fileclient/cc/jni_common.cc:583 Thread: 140354711938816 Could not find InodeAttribute constructor

 

Resolution

  1. Use the Studio MapR 4.1.0 libraries.

    Note: Studio v. 6.1.1 does not offer the 4.0.2 libraries, and most customers do not want to use a custom distribution to load them. The 4.1.0 libraries utilize the same version of Hadoop: 2.5.1. The 4.0.1 version of MapR libraries offered by Studio utilize an older version of Hadoop: 2.4.1. To avoid incompatibilities between the library versions, use the 4.1.0 MapR client against your 4.0.2 MapR cluster.

  2. In the MapR Job, add the following Hadoop properties:
    Property Value
    yarn.client.failover-proxy-provider org.apache.hadoop.yarn.client.MapRZKBasedRMFailoverProxyProvider
    mapreduce.jobhistory.intermediate-done-dir /var/mapr/cluster/yarn/rm/staging/history/done_intermediate
    mapreduce.jobhistory.done-dir /var/mapr/cluster/yarn/rm/staging/history/done
  3. You may continue to receive this error:
    ERROR JniCommon fs/client/fileclient/cc/jni_common.cc:583 Thread: 140354711938816 Could not find InodeAttribute constructor
  4. Use a tLibraryLoad to load the following library to the Job: maprfs-jni-4.0.2-mapr.jar.
  5. Use the library from 4.0.2 (as with the 4.1.0 library, there is still some incompatibility).
  6. To eliminate incompatibility between versions 4.0.2 and 4.1.0, replace the maprfs-4.1.0-mapr.jar with maprfs-4.0.2-mapr.jar.
  7. Run the Job.
Version history
Revision #:
5 of 5
Last update:
‎05-04-2017 07:21 PM
Updated by:
 
Labels (1)
Contributors
Tags (1)