topic Re: How to fetch files from directories based on dates in Talend Studio

How to fetch files from directories based on dates

shameer1 — Sat, 16 Nov 2024 05:28:54 GMT

Suppose I have few files in few directories as below.

/home/2019-06-23/test1.txt
/home/2019-06-23/test2.txt
/home/2019-06-24/test1.txt
/home/2019-06-24/test2.txt
/home/2019-06-25/test1.txt
/home/2019-06-25/test2.txt

/home/2019-06-26/test1.txt
/home/2019-06-26/test2.txt

My requirement is to fetch the files from those directory as per the two argument at run time i.e start_Dt and end_Dt.

If I pass Start_dt =2019-06-23 and end_Dt =2019-06-25 in my context variable then I should be fetching the below files only(from directory falling between the dates 2019-06-23 and 2019-06-25) and I need to append their context in one output file out.txt

/home/2019-06-23/test1.txt
/home/2019-06-23/test2.txt
/home/2019-06-24/test1.txt
/home/2019-06-24/test2.txt

/home/2019-06-25/test1.txt
/home/2019-06-25/test2.txt

How can i implement this in talend. I tried to use tLoop, but it seems to only work with number (like i=0;i<10;i++) and not with dates.

Re: How to fetch files from directories based on dates

lennelei — Wed, 26 Jun 2019 07:55:28 GMT

Hi,

number can be seen as the difference between two dates, therefore, you can use the tLoop in order to count the number of days between both dates for example.

for i=0 ; i<(date_end - date_start) days ; i++

you can use TalendDate routine to compute the number of days in the tLoop parameters :

From : 0
To : TalendDate.diffDate(TalendDate.parseDate("yyyy-MM-dd", "2019-06-25"),TalendDate.parseDate("yyyy-MM-dd", "2019-06-23"),"dd")
Step : 1

After that, you can use the CURRENT_VALUE from the tLoop to compute the date corresponding to the current loop :

TalendDate.addDate(TalendDate.parseDate("yyyy-MM-dd", "2019-06-23"), ((Integer)globalMap.get("tLoop_1_CURRENT_VALUE")), "dd")

Of course, you should set global variables to store the start and end dates; and you will have to format the date according to your folders format.

But personnaly, I would use the tLoop as a "while" loop :

Declaration : globalMap.put("moment", TalendDate.parseDate("yyyy-MM-dd", "2019-06-23"))
Condition : ((Date)globalMap.get("moment")).before(TalendDate.parseDate("yyyy-MM-dd", "2019-06-26"))
Iteration : globalMap.put("moment", TalendDate.addDate((Date)globalMap.get("moment"),1,"dd"))

Which is nice here is that you can simply use the global variable "moment" to access the current date :

TalendDate.formatDate("yyyy-MM-dd", (Date)globalMap.get("moment"))

Regards

Re: How to fetch files from directories based on dates

TRF — Wed, 26 Jun 2019 08:03:14 GMT

You can do it with just a tFileList to scan the /home content.

Then as folder names are well formed, you can use String.compareTo method to compare lexicographically each names with your context variables.

No need to deal with dates convertions in your case.

Re: How to fetch files from directories based on dates

shameer1 — Thu, 27 Jun 2019 07:08:18 GMT

Thanks lennelei and TRF for the response.

But can you please let me know which all components should be used to achieve this scenario. When I am using Run If link after tFileList then it is not traversing/Iterating to all the subdirectories. And if I use Iterate link after tFileList then i am not able to compare the start_dt and end_Dt with the subdirectories.

Can you please help me out.

Re: How to fetch files from directories based on dates

Loko — Thu, 04 Jul 2019 10:22:46 GMT

In a ESB route, Camel has seeded parameters to read files per date, see Camel File2 doc.

Re: How to fetch files from directories based on dates

lennelei — Tue, 09 Jul 2019 17:18:19 GMT

Hi,

tLoop --(iterate)--> tFileList --(iterate)--> [whatever you want to do with your files]

Regards