"hdp.version is not found" error when running Spark job under HDP in Studio

Talend Version (Required)       6.3.1 and earlier


hdp.version is not found error when running a Spark Job under HDP in Studio.
Additional Versions  
Product (Required) TBD
Component (Required) Studio
Problem Description

If you are running Spark under HDP, you may see the following error message:

hdp.version is not found,
Please set HDP_VERSION=xxx in spark-env.sh,
or set -Dhdp.version=xxx in spark.{driver|yarn.am}.extraJavaOptions
or set SPARK_JAVA_OPTS="-Dhdp.verion=xxx" in spark-env.sh
Problem root cause

HDP relies on the environment variable hdp.version in its various configuration files to support rolling upgrades. This can lead to issues when running Spark and MapReduce jobs in Talend Studio.

Solution or Workaround

Add the following properties into the Run > Spark Configuration > Advanced properties tab before running the Job.

Property: “spark.driver.extraJavaOptions”
Value: “-Dhdp.version=’xxxx’”
Property: “spark.yarn.am.extraJavaOptions”
Value: “-Dhdp.version=’xxxx’”
JIRA ticket number  
Version history
Revision #:
4 of 4
Last update:
‎09-29-2018 12:10 AM
Updated by:
Labels (1)