How to fetch files from directories based on dates

[shameer1]

Suppose I have few files in few directories as below.

/home/2019-06-23/test1.txt
/home/2019-06-23/test2.txt
/home/2019-06-24/test1.txt
/home/2019-06-24/test2.txt
/home/2019-06-25/test1.txt
/home/2019-06-25/test2.txt

/home/2019-06-26/test1.txt
/home/2019-06-26/test2.txt

My requirement is  to fetch the files from those directory as per the two argument at run time i.e start_Dt and end_Dt.

If I pass Start_dt =2019-06-23 and end_Dt =2019-06-25 in my context variable then I should be fetching the below files only(from directory falling between the dates  2019-06-23 and 2019-06-25) and I need to append their context in one output file out.txt

/home/2019-06-23/test1.txt
/home/2019-06-23/test2.txt
/home/2019-06-24/test1.txt
/home/2019-06-24/test2.txt

/home/2019-06-25/test1.txt
/home/2019-06-25/test2.txt

How can i implement this in talend. I tried to use tLoop, but it seems to only work with number (like i=0;i<10;i++) and not with dates.

[lennelei]

Hi,

number can be seen as the difference between two dates, therefore, you can use the tLoop in order to count the number of days between both dates for example.

for i=0 ; i<(date_end - date_start) days ; i++

you can use TalendDate routine to compute the number of days in the tLoop parameters :

From : 0
To : TalendDate.diffDate(TalendDate.parseDate("yyyy-MM-dd", "2019-06-25"),TalendDate.parseDate("yyyy-MM-dd", "2019-06-23"),"dd")
Step : 1

After that, you can use the CURRENT_VALUE from the tLoop to compute the date corresponding to the current loop :

TalendDate.addDate(TalendDate.parseDate("yyyy-MM-dd", "2019-06-23"), ((Integer)globalMap.get("tLoop_1_CURRENT_VALUE")), "dd")

Of course, you should set global variables to store the start and end dates; and you will have to format the date according to your folders format.

But personnaly, I would use the tLoop as a "while" loop :

Declaration : globalMap.put("moment", TalendDate.parseDate("yyyy-MM-dd", "2019-06-23"))
Condition : ((Date)globalMap.get("moment")).before(TalendDate.parseDate("yyyy-MM-dd", "2019-06-26"))
Iteration : globalMap.put("moment", TalendDate.addDate((Date)globalMap.get("moment"),1,"dd"))

Which is nice here is that you can simply use the global variable "moment" to access the current date :

TalendDate.formatDate("yyyy-MM-dd", (Date)globalMap.get("moment"))

Regards

[TRF]

You can do it with just a tFileList to scan the /home content.

Then as folder names are well formed, you can use String.compareTo method to compare lexicographically each names with your context variables.

No need to deal with dates convertions in your case.

[shameer1]

Thanks lennelei and TRF for the response.

But can you please let me know which all components should be used to achieve this scenario. When I am using Run If link after tFileList then it is not traversing/Iterating to all the subdirectories. And if I use Iterate link after tFileList then i am not able to compare the start_dt and end_Dt with the subdirectories.

Can you please help me out.

[Loko]

In a ESB route, Camel has seeded parameters to read files per date, see Camel File2 doc.

[lennelei]

Hi,

tLoop --(iterate)--> tFileList --(iterate)--> [whatever you want to do with your files]

Regards