One Star

Talend spark job in cluster mode

How can talend spark job be executed in cluster mode? Talend supports local, stand alone and yarn-client mode. I want to run it in yarn cluster mode. Is it possible?
3 REPLIES
Moderator

Re: Talend spark job in cluster mode

Hi,
We have redirected your issue to our bigdata experts and then come back you as soon as we can.
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Employee

Re: Talend spark job in cluster mode

Hi,
The yarn-cluster mode is currently not supported in the studio.
Regards,
Rémy.
One Star

Re: Talend spark job in cluster mode

Is there any way I can take the export, change the batch file / shell file to achieve it? I know that we can pass deploy mode (yarn-client vs yarn-cluster) as an argument to the spark-submit shell command. 
Can the export shell file be modified to achieve the same?
./bin/spark-submit \
--class <main-class> \
--master <master-url> \
--deploy-mode <deploy-mode> \
--conf <key>=<value> \
... # other options
<application-jar> \