topic Re: Submitting Spark Job using Talend tSSH component not working as requir in Talend Studio

Submitting Spark Job using Talend tSSH component not working as requir

Anonymous — Tue, 24 Nov 2015 12:30:33 GMT

I have created and spark job and when I running that job directly from console its taking 5 min to completed but when I am submitting the same job using Talend tSSH component its taking around 2 Hours and not getting finished.
Please help me on the above.

Re: Submitting Spark Job using Talend tSSH component not working as requir

Anonymous — Wed, 25 Nov 2015 08:04:11 GMT

Hi,

when I am submitting the same job using Talend tSSH component

Is there any error message printed on console? How did you set your tSSH component?
Best regards
Sabrina

Re: Submitting Spark Job using Talend tSSH component not working as requir

Anonymous — Wed, 25 Nov 2015 10:37:24 GMT

It Is not throwing any error in console and job.Attached configuration for tSSH component where we are giving Hostname,Username and password via context variables and in the commands string we are using the spark-submit command.

Re: Submitting Spark Job using Talend tSSH component not working as requir

Anonymous — Wed, 25 Nov 2015 10:38:48 GMT

configuration screen shoot

Re: Submitting Spark Job using Talend tSSH component not working as requir

_AnonymousUser — Mon, 27 Feb 2017 10:40:10 GMT

Did you found a solution for that? I am facing the same issue.
Best,
Marlena

Re: Submitting Spark Job using Talend tSSH component not working as requir

Anonymous — Thu, 03 May 2018 09:39:06 GMT

Hi Sabrina,

This issue is occurring for me too. The job is getting stuck and we are able to see it stuck in "running process in linux" for hours. It is not getting killed but the control is not getting reverted to Talend when executing spark-submit from TSSH. Initial analysis suggests issue with logging streaming logs.
Thanks,
Akash Jaiswal