Need help with what to do with previous input files in case of script failure.

[JSey]

Hi everyone

I have a Talend script that will retrieve the latest csv files from a sftp server (let's say filename abc-mmDDYYYY with mmDDYYYY being the date). I have a tfilelist to use a mask based on the file names and to sort the files by modified date desc. I need feedback on best practices after this part:

1) Assuming the files are sent to the sFTP daily, if the script doesn't run for 2 days, it will have 2 set of files. How can it process both set of files? I did a test by making a copy of one file and the script only processed one set of file.

 

2) Would it be best practice to move the script to a processed folder on the sftp (or your suggestion)?

Thank you for your help everyone

[JSey]

Any help would be appreciated, thanks.