Talend spark job in cluster mode

One Star

Talend spark job in cluster mode

How can talend spark job be executed in cluster mode? Talend supports local, stand alone and yarn-client mode. I want to run it in yarn cluster mode. Is it possible?
Moderator

Re: Talend spark job in cluster mode

Hi,
We have redirected your issue to our bigdata experts and then come back you as soon as we can.
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Employee

Re: Talend spark job in cluster mode

Hi,
The yarn-cluster mode is currently not supported in the studio.
Regards,
Rémy.
One Star

Re: Talend spark job in cluster mode

Is there any way I can take the export, change the batch file / shell file to achieve it? I know that we can pass deploy mode (yarn-client vs yarn-cluster) as an argument to the spark-submit shell command. 
Can the export shell file be modified to achieve the same?
./bin/spark-submit \
--class <main-class> \
--master <master-url> \
--deploy-mode <deploy-mode> \
--conf <key>=<value> \
... # other options
<application-jar> \
Six Stars

Re: Talend spark job in cluster mode

Hi Sankalp,

Did you find a way to run Talend job jar over a cluster by spark-submit ?