topic Re: How to process 2 latest file from a folder based on modified date in Talend Studio

How to process 2 latest file from a folder based on modified date

Anonymous — Sat, 16 Nov 2024 05:04:09 GMT

I have a scenario where I will have some set of files which gets created by some external system for every 10 minutes or so.How to process only the 2 latest file from all the files present in the folder sorted on file creation date/modified date in desc?

Ex : C:/Sample/Incoming_Files -> Emp_YYYYMMDDHHMMSS is the file format

Emp_20190801001534.csv

Emp_20190801041522.csv

Emp_20190801031555.csv

Emp_20190801001504.csv

Emp_20190801101517.csv

How to process the 2 latest file from the folder based om modified date?

Could someone please make it quick?

Thanks,

Kiran Kumar

Re: How to process 2 latest file from a folder based on modified date

Ganshyam — Tue, 06 Aug 2019 10:25:23 GMT

Hello,

Make use of tfileproperties which will give you details of the modified date and time of file.

Thanks

Ganshyam

Re: How to process 2 latest file from a folder based on modified date

Anonymous — Tue, 06 Aug 2019 11:28:42 GMT

There are a few steps to this. I have created an example job which can be seen below....

First of all, use a tFileList to monitor your folder and retrieve CSV files. Using this component is described in the documentation.

The next is a tJavaFlex. Create two columns for this; a filedate column (Date) and a filepath column (String). You need to convert the filename to a date and pass the filepath as normal. This is done in the MainCode section. The code I have used is below....

String filename = ((String)globalMap.get("tFileList_1_CURRENT_FILE"));
filename = filename.substring(4, filename.length()-4);

row1.filedate = routines.TalendDate.parseDate("yyyyMMddHHmmss", filename);
row1.filepath = ((String)globalMap.get("tFileList_1_CURRENT_FILEPATH"));

After that, the records are sent to a tHashOutput set with the "Append" option on.

In the next subjob, you start with a tHashInput component. This is linked to the tHashOutput and has the same schema.

Join that to a tSortRow component. My config is shown below...

This sorts the files by date.

The next component is the tMap. This is used to filter only the two newest files. I do this by creating a "count" record in a tMap variable. This use the Numeric.sequence method. As the records pass through the tMap, they are assigned a number. The tMap output then filters the records which are allowed to be output to have a count less than 3. This can be seen below...

After this, I have a tLogRow simply to show the result.

Note: I have just realised that the example is set up for the two oldest files. Just switch the tSortOrder component's sort order to asc to get the two newest.