topic tFileExist not waiting for tSSH to complete in Talend Studio

tFileExist not waiting for tSSH to complete

Anonymous — Thu, 29 Aug 2019 12:32:05 GMT

Our client recently started using CloudFlare protection and it blocked the use of tFileFetch. I am able to work around this by using cURL in a tSSH component.

I am iterating through a file list and passing the file names in the form of a variable to a tSSH component to do the curl to download the files from a website and then I use tFileExist to check that the file downloaded successfully to populate another variable which serves as a flag to let my job know when all required files are downloaded.

What I get is that the tFileExist component is not waiting until the file is done downloading before executing. I have tried fix this by adding a IF condition which checks the exit code of the tSSH ( ((Integer)globalMap.get("tSSH_2_EXIT_CODE")) == 0 ) but this does not help.

I think what might be happening is that the tSSH is sending the success code before finishing the file download.

How can I handle this? Any help would be much appreciated.

Re: tFileExist not waiting for tSSH to complete

Anonymous — Thu, 29 Aug 2019 16:03:34 GMT

Can you post a screenshot of your job?

Re: tFileExist not waiting for tSSH to complete

Anonymous — Fri, 30 Aug 2019 08:41:57 GMT

Here is the relevant part of the job. If I put in a sleep component instead of the selected IF then the Check happens as expected after the file is done. But if a very large file or slow internet causes it to not finish the download then it will obviously fail.

Re: tFileExist not waiting for tSSH to complete

Anonymous — Fri, 30 Aug 2019 09:09:27 GMT

Hmmmm, this looks like the terminal response is indicating the command has finished while the file is still downloading. A good way of resolving this would be to add a tLoop with your tFileExist component. Set a globalMap variable of "found" to be set to false. Set the tLoop's logic to fire while "found" is false. When the tFileExist returns true, set "found" to be true.

Essentially the flow after the tSSH would be .....

tJava (to set the boolean "found" globalMap)

(iterate link)

tLoop (with a while loop based on the "found" variable)

(iterate link)

tFileExist

(run if link based on the tFileExist result)

tJava (to set "found" to true)

Then you would trigger the rest of the code from the tLoop.

Re: tFileExist not waiting for tSSH to complete

Anonymous — Fri, 30 Aug 2019 09:15:06 GMT

Thank you for your suggestion. I will try and see if it works!