Big Data Batch Job Spark Configuration error

One Star

Big Data Batch Job Spark Configuration error

I am trying to run a big data batch job on spark and it is giving an exception to add hdp version in spark-env.sh I am using horton works HDP 2.5 distribution and added that to spark-env.sh file but I am still getting the same error. PFA of my error.
Seven Stars

Re: Big Data Batch Job Spark Configuration error

Make sure to specify the following arguments in the spark configuration when using YARN (using HDP)
spark.driver.extraJavaOptions=""
spark.yarn.am.extraJavaOptions=""
spark.hadoop.mapreduce.application.framework.path=""
spark.hadoop.mapreduce.application.classpath=""

Re: Big Data Batch Job Spark Configuration error

Thanks for the reply. Can you please explain me in detail what to add in that arguments in quotes. Like I have added this already in my job
Seven Stars

Re: Big Data Batch Job Spark Configuration error

You can usually find these in your YARN config in Ambari:
Example below where 2.3.2.0-2950 is your HDP version
spark.driver.extraJavaOptions="-Dhdp.version=2.3.2.0-2950"
spark.yarn.am.extraJavaOptions="-Dhdp.version=2.3.2.0-2950"
spark.hadoop.mapreduce.application.framework.path="$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:$PWD/mr-framework/hadoop/share/hadoop/tools/lib/*:/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure"
spark.hadoop.mapreduce.application.classpath="/hdp/apps/2.3.2.0-2950/mapreduce/mapreduce.tar.gz#mr-framework"

What’s New for Talend Spring ’19

Watch the recorded webinar!

Watch Now

Tutorial

Introduction to Talend Open Studio for Data Integration.

Watch

Downloads and Trials

Test drive Talend's enterprise products.

Downloads

Definitive Guide to Data Integration

Practical steps to developing your data integration strategy.

Download