Do not input private or sensitive data. View Qlik Privacy & Cookie Policy.
Skip to main content

Announcements
Join us to spark ideas for how to put the latest capabilities into action. Register here!
cancel
Showing results for 
Search instead for 
Did you mean: 
shameer1
Contributor II
Contributor II

How to fetch files from directories based on dates

Suppose I have few files in few directories as below.

 

/home/2019-06-23/test1.txt
/home/2019-06-23/test2.txt
/home/2019-06-24/test1.txt
/home/2019-06-24/test2.txt
/home/2019-06-25/test1.txt
/home/2019-06-25/test2.txt

/home/2019-06-26/test1.txt
/home/2019-06-26/test2.txt

 

My requirement is  to fetch the files from those directory as per the two argument at run time i.e start_Dt and end_Dt.

 

If I pass Start_dt =2019-06-23 and end_Dt =2019-06-25 in my context variable then I should be fetching the below files only(from directory falling between the dates  2019-06-23 and 2019-06-25) and I need to append their context in one output file out.txt

 

/home/2019-06-23/test1.txt
/home/2019-06-23/test2.txt
/home/2019-06-24/test1.txt
/home/2019-06-24/test2.txt

/home/2019-06-25/test1.txt
/home/2019-06-25/test2.txt

 

How can i implement this in talend. I tried to use tLoop, but it seems to only work with number (like i=0;i<10;i++) and not with dates.

Labels (2)
5 Replies
lennelei
Creator III
Creator III

Hi,

 

number can be seen as the difference between two dates, therefore, you can use the tLoop in order to count the number of days between both dates for example.

for i=0 ; i<(date_end - date_start) days ; i++

you can use TalendDate routine to compute the number of days in the tLoop parameters :

From : 0
To : TalendDate.diffDate(TalendDate.parseDate("yyyy-MM-dd", "2019-06-25"),TalendDate.parseDate("yyyy-MM-dd", "2019-06-23"),"dd")
Step : 1

0683p000009M5j9.png

After that, you can use the CURRENT_VALUE from the tLoop to compute the date corresponding to the current loop :

TalendDate.addDate(TalendDate.parseDate("yyyy-MM-dd", "2019-06-23"), ((Integer)globalMap.get("tLoop_1_CURRENT_VALUE")), "dd")

Of course, you should set global variables to store the start and end dates; and you will have to format the date according to your folders format.

 

 

But personnaly, I would use the tLoop as a "while" loop :

Declaration : globalMap.put("moment", TalendDate.parseDate("yyyy-MM-dd", "2019-06-23"))
Condition : ((Date)globalMap.get("moment")).before(TalendDate.parseDate("yyyy-MM-dd", "2019-06-26"))
Iteration : globalMap.put("moment", TalendDate.addDate((Date)globalMap.get("moment"),1,"dd"))

0683p000009M5jE.png

Which is nice here is that you can simply use the global variable "moment" to access the current date :

TalendDate.formatDate("yyyy-MM-dd", (Date)globalMap.get("moment"))

 

Regards

TRF
Champion II
Champion II

You can do it with just a tFileList to scan the /home content.

Then as folder names are well formed, you can use String.compareTo method to compare lexicographically each names with your context variables.

No need to deal with dates convertions in your case.

shameer1
Contributor II
Contributor II
Author

Thanks lennelei and TRF for the response.

But can you please let me know which all components should be used to achieve this scenario. When I am using Run If link after tFileList then it is not traversing/Iterating to all the subdirectories. And if I use Iterate link after tFileList then i am not able to compare the start_dt and end_Dt with the subdirectories.

 

Can you please help me out.

Loko
Creator II
Creator II

In a ESB route, Camel has seeded parameters to read files per date, see Camel File2 doc.

lennelei
Creator III
Creator III

Hi,

tLoop --(iterate)--> tFileList --(iterate)--> [whatever you want to do with your files]

Regards