Submitting Spark Job using Talend tSSH component not working as requir

Submitting Spark Job using Talend tSSH component not working as requir

I have created and spark job and when I running that job directly from console its taking 5 min to completed but when I am submitting the same job using Talend tSSH component its taking around 2 Hours and not getting finished.
Please help me on the above.
Moderator

Re: Submitting Spark Job using Talend tSSH component not working as requir

Hi,
when I am submitting the same job using Talend tSSH component

Is there any error message printed on console? How did you set your tSSH component?
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.

Re: Submitting Spark Job using Talend tSSH component not working as requir

It Is not throwing any error in console and job.Attached configuration for tSSH component where we are giving Hostname,Username and password via context variables and in the commands string we are using the spark-submit command.

Re: Submitting Spark Job using Talend tSSH component not working as requir

configuration screen shoot
One Star

Re: Submitting Spark Job using Talend tSSH component not working as requir

Did you found a solution for that? I am facing the same issue.
Best,
Marlena
One Star

Re: Submitting Spark Job using Talend tSSH component not working as requir

Hi Sabrina,

This issue is occurring for me too. The job is getting stuck and we are able to see it stuck in "running process in linux" for hours. It is not getting killed but the control is not getting reverted to Talend when executing spark-submit from TSSH. Initial analysis suggests issue with logging streaming logs.
Thanks,
Akash Jaiswal