How to loop through the files in the subfolder directory in talend java using tmap?get only recent day file

[Haresh]

I am looking to download files from the ftp site using tftpfilelist component and i dont see an option to check for "include subdirectories" unlike tfilelist.

My job flow looks like this:

I need to find the latest files from last 1 day and download using tftpget.

Since there are lot of files in the remote directory of the sftp site,i am using tmap function after iterating through the tbufferoutput/input component.

The tmap function looks like this: TalendDate.diffDate(TalendDate.parseDate("yyyy-MM-dd",TalendDate.getDate("yyyy-MM-dd")),TalendDate.parseDate("EEE MMM dd HH:mm:ss zzz yyyy",row8.mtime_string)) <1

The row8.mtime_string indicates the last modified date of the file in ftp site using tfileproperties and if it is less than 1 day timeperiod,then download it.

However after iterating through all the folders it errors at tmap with output as NULL?

I cant seem to get within the subfolders to scan through the files at the ftp site ,please suggest

The ftp site may look like this with folders : 

filesearch1->downloadfile.txt

downloadfile.txt is the one i would want to scan and download using tftpget.But i cannot go within the subfolders ,using tmap?enter

[Anonymous]

Hi

Refer to this topic, use two tFTPFileList components to achieve it, the first tFTPFileList is set to the parent folder without filemask, the second tFTPFileList is set to iterate each record set ( subfolders found by the first tFTPFileList).

Regards

Shong

[Haresh]

Thanks @Shicong Hong​  that is similar to what i had done but the problem here lies with the fact that i am using tmap to filter records for only the previous day and in the ftp site there are folders available since 2019 and so around 1000 folders each having 25 files ,if i were to loop through all of them it would be like 25000 files to process and then go into ->tbufferedoutput->tbufferedinput->tmap (filter for last one day) to fetch the recent files which is drastically time consuming almost.

Is there a better way to handle this?

[Haresh]

The question remains if there is any efficient way to directly get the file folder last modified date using tftpfileproperties component and only iterate through those subfolders alone rather than having to go through all of them at once

And also how to download the folder that could have 25 files inside it using tftpget?