Runtime exception when running a Spark Job with MapR-DB components

Talend Version (Required)       6.3.1

Summary

 
Additional Versions  
Product (Required) Talend Big Data
Component (Required) Spark MapR
Problem Description

When running a Talend 6.3.1 Spark Job (in YARN mode) that uses MapR-DB components against MapR 5.1, it fails with the following exception:

java.io.IOException: java.lang.RuntimeException: Error occurred while instantiating com.mapr.fs.hbase.MapRTableMappingRules.

And the following Java Exception Stack:

Error occurred while instantiating com.mapr.fs.hbase.MapRTableMappingRules.
==> org/apache/hadoop/hbase/client/mapr/BaseTableMappingRules.
at test3.testspark_0_1.TestSpark$row12StructInputFormat_tHBaseInput_1.configure(TestSpark.java:791)
...
Caused by: java.io.IOException: java.lang.RuntimeException: Error occurred while instantiating com.mapr.fs.hbase.MapRTableMappingRules.
==> org/apache/hadoop/hbase/client/mapr/BaseTableMappingRules.
at org.apache.hadoop.hbase.client.mapr.TableMappingRulesFactory.create(TableMappingRulesFactory.java:68)
at org.apache.hadoop.hbase.client.HTable.initIfMapRTableImpl(HTable.java:475)
Problem root cause

With MapR 5.1 or 5.2, this issue is because the HBase/MapR-DB classpath is missing in the spark.hadoop.yarn.application.classpath parameter that the Talend Spark Job is using by default. Without additional classpath configuration on the Spark configuration tab of the Job, the HBase/MapR-DB Job cannot successfully find the required classes and packages in the cluster.

Solution or Workaround

The solution consists of providing the needed HBase/MapR-DB libraries for the Spark Job execution.

For instructions on how to implement this, see Adding HBase classpath in the Talend Help Center.

JIRA ticket number  
Version history
Revision #:
10 of 10
Last update:
‎11-02-2017 02:16 PM
Updated by:
 
Contributors
Tags (2)