How to retrieve the last 8 files from F

Nawaz26893 — Sat, 16 Nov 2024 00:16:27 GMT

Hello everyone,

I have a Talend job which will do the following:

Grab the most recent 4 files from an FTP server and copy them over to my Talend Server for processing. The files are uploaded one everyday
I am using FTPList to see all the different files and dates associated with them. At this moment. I have used a tLogRow to print out the details from the global variables using Iteratetoflow on my TfileList component.

What is the best way to move the most recent 4 files from my FTP server to Talend Server for processing? Also:

Once the files are in the Talend Server I want to process the files based on todays date. So for instance the filename is XYZ_20210410.csv I would need some type of filemask that would (A) be updated on a daily basis so the filemask would become tomorrow *20210411.csv for instance so it grabs the file dated for today from the 8 files it grabbed initially from the FTP Server.
The file would then be processed and some sql queries would be run to push this into a postgres DB which I have built the job for.
I would then like to iterate over the next 3 files and reprocess them just in case there was an issue to the processing the previous day.

My main issues are how to grab only the last 4 files based on recency and then processing the files by date using a mask on their names as they also have a date. I think I would need to have a logic that is able to store the filenames date and compare it to a global variable whose date gets updated daily and is saved as a context variable and then compare and run the job if they match. Any help would be super appreciated as I am very new to this amazing tool.

Thank you all.

Re: How to retrieve the last 8 files from F

MS5 — Mon, 12 Apr 2021 15:58:28 GMT

Hello,

For the current date, you can use function TalendDate.getCurrentDate() that you will store in a global variable.

By using components tFTPFileList with a link type "Iterate", you can access all files name by using the global variable CURRENT_FILE (created automatically by the compoent tFTPFileList).

Then you can filter using a tFilter one this filename by comparing the global var that contain current date and the date in the name of the file.

Of course you have to transform each date (current date and file date) to the good format to be sure that the comparision will be ok.

You will have something like :

tFTPFileList -> (iterate) tFixedFlowInput with one column = CURRENT_FILE globalVar -> tFilter -> your list of 4 files.

topic How to retrieve the last 8 files from F in Talend Studio

How to retrieve the last 8 files from F

Re: How to retrieve the last 8 files from F