topic Re: How to transfer files (sequentially) from FTP directory in Talend Studio

How to transfer files (sequentially) from FTP directory

_AnonymousUser — Mon, 20 Apr 2015 13:30:48 GMT

Hi all,
I'm conducting a review of Talend capabilities through some real life scenarios for my company's Data Engineering team.
The case I'm finding hard to resolve is the scenario where you have several zipped(.gz) files in an FTP directory, and you want to transfer them one by one to an Oracle DB table.
I have configured the most part of the job, but the tricky part is that one file has to be transferred each time and since the file is transferred to be ignored from the job in the next run. (not to transfer data that have already been inserted to the DB table).
These files are received daily and have a similar file name pattern , like TRANSACTIONS_YYYYMMDD, with the date increasing accordingly each day the file is received.
Is there any out of the box component to perform this process?
I would appreciate any help.
Thank you in advance

Re: How to transfer files (sequentially) from FTP directory

_AnonymousUser — Mon, 20 Apr 2015 13:38:07 GMT

Additional note: Forgot to mention that I cannot edit any file on the FTP folder, so I can't run the scenario of copy the file to a different folder and start the transfer from there and then deleting it from the original folder when the processing has ended successfully.

Re: How to transfer files (sequentially) from FTP directory

Anonymous — Mon, 20 Apr 2015 15:36:33 GMT

hi,
first retrieve all files from remote ftp repository.(tftpfilelist)
Create a table to save which file has already been loaded & push to Oracle DB (with a status field).
Really upload files that aren't with status 'OK' (mean files already uploaded and pushed into DB with success).

hope it helps
laurent

Re: How to transfer files (sequentially) from FTP directory

Anonymous — Mon, 20 Apr 2015 15:44:56 GMT

could be something like :

regards
laurent

Re: How to transfer files (sequentially) from FTP directory

_AnonymousUser — Tue, 21 Apr 2015 10:07:05 GMT

Thank you very much Laurent.
I will try this solution.

Re: How to transfer files (sequentially) from FTP directory

Anonymous — Tue, 21 Apr 2015 10:29:23 GMT

tell us if it's resolved
regards