topic Re: Error and missing file while downloading using SFTP in Talend Studio

Error and missing file while downloading using SFTP

ars — Fri, 20 May 2022 12:58:29 GMT

Hi all,

I've developed a job in order to download files from an SFTP server. It's being up and running for many years and I didn't have any problem in the past.

A special notice to consider, is that the SFTP server is hosted on external partner prremises and according to its configuration, files are being deleted after the downloading is completed.

So, everything work fine so far but recently I'm facing an unknown (and strange!) issue.

Under unspecified circumstances, component tFTPGet throws the error message {com.jcraft.jsch.SftpException} and the job is terminated.

But while the file has not been uploaded (not even partially) the SFTP server considers that was downloaded and the file is removed from it. So, if I try to re-download it, it's not there!!!

Any thoughts, why the Talend could crash and the file is being lost??

Best regards,

Ars

Re: Error and missing file while downloading using SFTP

Anonymous — Mon, 23 May 2022 08:32:51 GMT

Hello,

By design, tFTPGet will look at your local drive (i.e. C drive, C:/ASDG/DEX) to place/store your file , so are you using local directory instead of remote directory in it?

Could you please post the full error stack trace here and send us a snapshot of what you are seeing or trying to do so we can better understand the issue.

And then I will look into your issue as soon as possible and get back to you.

Best regards

Sabrina

Re: Error and missing file while downloading using SFTP

ars — Wed, 25 May 2022 12:48:42 GMT

Hello,

Thank you for your reply.

Basically, my tFTPGet component uses a local network drive, to store the downloaded files:

Regarding the full error stack trace, unfortunetally the only error I can capture is through log (tLogCatcher info).

This is because the job is executed automatically via central pc and at the moment (when) it fails I have no view of the console.

Also, I can't reproduce the problem 'ad hoc' because the SFTP where we download the files is on an external partner and I have no rights to upload a file. But mostly, even I had the right to do it, it might be impossible finally to reproduce the error. Because, the exception occurs some times and this is what we need to find. The reason of why this is happening.

This is what tLogCather, exports on log table:

There is no any other 'hidden' information on the field. Only this value: com.jcraft.jsch.SftpException:

Any thoughts?

BR,

Ars

Re: Error and missing file while downloading using SFTP

Anonymous — Thu, 26 May 2022 07:19:28 GMT

Hello,

We can see that you are using a relative path as your local directory. There is access to this shared network drive? And It does work well when get files from FTP server?

In addition, if you don't mind, could you please open the error log generated automatically by the Studio to see whether you have error stack trace there; if you do, we will appreciate it a lot if you can paste it here.

About how to open this error log: in your Studio, click Window to open the menu, then select Show View->General->Error Log. Once you see an error, double click it.

Best regards

Sabrina

Re: Error and missing file while downloading using SFTP

ars — Fri, 27 May 2022 09:47:47 GMT

Hello,

Yes, job has full access on the shared folder and it does work well when we get files from the SFTP server.

Now, regarding the log you need, as I mentioned the job is executed automatically via central pc.Thus, at the moment of execution there no any Studio instance open and only the .bat (from the deployment package) file is executed.

And I can't reproduce the error from my Studio instance; this is something that is happening radomly from the automated job.

Is there any other log info available from the 'runtime' job?

BR,

Ars

Re: Error and missing file while downloading using SFTP

Anonymous — Mon, 30 May 2022 03:40:02 GMT

Hello,

Do you mean this FTP job can be executed successfully in studio? What's your runtime server for .bat file?CMD? Window scheduler?

Best regards

Sabrina

Re: Error and missing file while downloading using SFTP

ars — Mon, 30 May 2022 06:46:48 GMT

Hello,

The FTP job executed succefully in studio. On runtime server it's executed (most times succefully) using the .bat file, which is triggered via Window scheduler.

BR,

Ars

Re: Error and missing file while downloading using SFTP

Anonymous — Mon, 30 May 2022 07:50:49 GMT

Hello,

As this job can be executed successfully in talend studio, I'm supposing something be wrong with Windows task scheduler for using the mapped disk drive letters.

Have you tried to add some debugging code in your job .bat file to see if there is any issue?

Best regards

Sabrina

Re: Error and missing file while downloading using SFTP

ars — Mon, 30 May 2022 08:51:15 GMT

Hello,

The windows scheduler is working well. The job is executed on daily basis and most times it's completed successfully.

So this is not the point, of our investigation! The question is why the tFTPGET fails in some cases and raises the SFTP exception.

BR,

Ars

Re: Error and missing file while downloading using SFTP

Anonymous — Tue, 31 May 2022 07:12:11 GMT

Hello,

Do you have any record when tFTPGET fails? What's kind of some cases? Specific environment cases?

Is there any possible or detailed steps for us to repro this issue?

Best regards

Sabrina

Re: Error and missing file while downloading using SFTP

ars — Tue, 31 May 2022 08:10:58 GMT

Hello,

The only record I'm getting when tFTPGet fails, is this one via tLogCatcher: {com.jcraft.jsch.SftpException} and the job is terminated.

You can't repro the issue, because you need to establish connection to the SFTP, but I can't provide the details for that.

Job is very simple. A connection is established, next the get component start the downloading. And under unspecified circumstances, some times this is failed. So, to my pespective, is there any suggestion why tFTPGet could failed? And if so is there any workaround?

BR,

Ars

Re: Error and missing file while downloading using SFTP

Anonymous — Tue, 31 May 2022 08:22:54 GMT

Hello,

As this is a random issue, we are not able to give a solution/ workaround till we can find out the root cause.

Would you mind creating a jira issue on talend bug tracker?

https://jira.talendforge.org/login.jsp

Best regards

Sabrina

Re: Error and missing file while downloading using SFTP

ars — Tue, 31 May 2022 09:05:46 GMT

Hello,

Ok, understood. Thank you for your time.

BR,

Ars

Re: Error and missing file while downloading using SFTP

Hamza011 — Wed, 31 Jul 2024 12:58:20 GMT

hello , any support for the issue, i face the same problem !!!

Re: Error and missing file while downloading using SFTP

Xiaodi_Shi — Thu, 01 Aug 2024 03:31:04 GMT

Hello @Hamza011

I replied to your another topic about:

https://community.qlik.com/t5/Design-and-Development/com-jcraft-jsch-JSchException-Auth-cancel-for-methods-password/m-p/2474056#M141083

Could you please take a look at it to see if it helps?

Best regards

Sabrina