Four Stars

Spark as YARN Client

Hello everyone,
I'm currently managing a Hortonworks HDP cluster and I'm testing Talend Big Data licence evaluation.
I would like to test the Spark components with my cluster.
When I set in the Spark connection the "YARN Client" client, the component is asking me a .jar as configuration file.
Do you know where I can find/generate such a jar file?
Thank you.
Regards,
Orlando
3 REPLIES
Moderator

Re: Spark as YARN Client

Hi Orlando,
When I set in the Spark connection the "YARN Client" client, the component is asking me a .jar as configuration file.

For this field, you need to browse to the local jar file that contains the configuration of the Yarn service to be used.
The configuration files that must present in this jar file are: core-site.xml, hdfs-site.xml, mapred-site.xml, yarn-site.xml.
 Note: You can press F1 (focus your mouse on tSparkConnection)to find the related component reference of tSparkConnection.

Best regards
Sabrina


--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: Spark as YARN Client

Hi,
Even i have tried building the jar file but its not working for me. Any luck for you?
If so could you please help me how to build the jar file??
I have these 4 xml files on the edge node. i packaged them in jar file. But its not working. Anything else i have to do?
Regards,
Shanker
Four Stars

Re: Spark as YARN Client

Hi Shanker,
I didn't manage to get it work. I'm managing a Hortonwork's distribution of Hadoop.
I had to change ${hdp.version} to 2.2.4.2-2 (or your version here) in the mapred-site.xml.
Now my job is deployed but failed after a few seconds with this error :
Exception in thread "main" java.lang.IllegalArgumentException: Invalid ContainerId: container_e157_1443206132389_0129_01_000001
...
Caused by: java.lang.NumberFormatException: For input string: "e157"
...
... 11 more
I'm still looking for a solution.
Cheers,
Orlando