[resolved] Error in Spark job run via YARN Client

Seven Stars

[resolved] Error in Spark job run via YARN Client

Hello, I've create a Spark job and it runs fine in local mode through Talend, but when I try to use YARN Client mode I get the following error. It isn't clear what property in Talend might be configured incorrectly.  Spark applications run outside of Talend do not have this issue.  Has anyone else run into this?
java.lang.ClassCastException: org.apache.spark.deploy.SparkHadoopUtil cannot be cast to org.apache.spark.deploy.yarn.YarnSparkHadoopUtil
at org.apache.spark.deploy.yarn.YarnSparkHadoopUtil$.get(YarnSparkHadoopUtil.scala:163)
at org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:257)
at org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:561)
at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:115)
at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:59)
at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:141)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:497)
at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:61)
at local_project.project_demo_0_1.project_demo.runJobInTOS(project_demo.java:1985)
at local_project.project_demo_0_1.project_demo.main(project_demo.java:1868)

Accepted Solutions
Seven Stars

Re: [resolved] Error in Spark job run via YARN Client

I worked further with Talend support to add additional parameters to the spark configuration to get past this issue.
I had to set my HDP version with these configs:
spark.driver.extraJavaOptions
spark.yarn.am.extraJavaOptions
In addition, set these: 
spark.hadoop.mapreduce.application.framework.path
spark.hadoop.mapreduce.application.classpath

All Replies
Moderator

Re: [resolved] Error in Spark job run via YARN Client

Hi,
Could you please indicate on which build version you got this issue? Screenshots of job setting will be helpful for us to address your issue.
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Seven Stars

Re: [resolved] Error in Spark job run via YARN Client

I worked further with Talend support to add additional parameters to the spark configuration to get past this issue.
I had to set my HDP version with these configs:
spark.driver.extraJavaOptions
spark.yarn.am.extraJavaOptions
In addition, set these: 
spark.hadoop.mapreduce.application.framework.path
spark.hadoop.mapreduce.application.classpath