Big Data Batch Job Spark Configuration error

One Star

Big Data Batch Job Spark Configuration error

I am trying to run a big data batch job on spark and it is giving an exception to add hdp version in I am using horton works HDP 2.5 distribution and added that to file but I am still getting the same error. PFA of my error.
Seven Stars

Re: Big Data Batch Job Spark Configuration error

Make sure to specify the following arguments in the spark configuration when using YARN (using HDP)

Re: Big Data Batch Job Spark Configuration error

Thanks for the reply. Can you please explain me in detail what to add in that arguments in quotes. Like I have added this already in my job
Seven Stars

Re: Big Data Batch Job Spark Configuration error

You can usually find these in your YARN config in Ambari:
Example below where is your HDP version

What’s New for Talend Spring ’19

Watch the recorded webinar!

Watch Now


Introduction to Talend Open Studio for Data Integration.


Downloads and Trials

Test drive Talend's enterprise products.


Definitive Guide to Data Integration

Practical steps to developing your data integration strategy.