How to get the last 2 days file from the remote ftp site?

[Haresh]

I am trying to retrieve the last 2 days worth of files from the remote ftp site when they have 5000 odd files present in the sftp site.I just want to sort the file list by desc order and get the last 2 days files alone into the local folder.

i am thinking if there is any way to implement it without using tbufferoutput and iterating through all the list of the files(5000) in the ftp site.Is there a quicker way to do this ,please help.

Please see the job flow below:

i am trying to read the ftpfile properties like the date and sort by desc and give the range and fetch the latest 2 days only.

Urgently need a better solution

[Anonymous]

Hi

I have done similar needs, in my experience, there is no other better way.

Regards

Shong

[Haresh]

Thanks shong for the response.However i am trying to write java code as below,but i am getting null value in the console for the output...any idea what i am missing here so that file details populate?

tjava goes like this:

Date lastModifiedDate = TalendDate.parseDate("EEE MMM dd HH:mm:ss zzz yyyy", input_row.mtime_string);

Date current_date = TalendDate.getCurrentDate();

System.out.println(lastModifiedDate);

System.out.println(current_date);

System.out.println(((String)globalMap.get("tFTPFileList_1_CURRENT_FILE")));

if(TalendDate.diffDate(current_date, lastModifiedDate,"dd") <= 1) {

output_row.abs_path = input_row.abs_path;

System.out.println(output_row.abs_path);

}

tlogrow3:

[Haresh]

@Shicong Hong​  even if that is the case,does it take a lot of time for the files to loop through all the 5000 files in the ftp folder and scan through and perform the sort?

I was wondering atleast without scanning/iterating through the entire 30000 files it would be best if i could directly pull only the recent ones without having to iterate through the entire list and then later sort->validate the results?

[houssem_B]

you can filter them using tmap filter like this

[Haresh]

Thanks @houssem Boufaden​  i tried your solution but i am running into an error as below:

I am using expression builder to parse the mtime_string :

TalendDate.diffDate(TalendDate.getCurrentDate(),TalendDate.parseDate("yyyy-MM-dd",mtime_string),"dd") <= 2

On the same note,i tried a similar tjava code but it does not work properly:

Date lastModifiedDate = TalendDate.parseDate("EEE MMM dd HH:mm:ss zzz yyyy", row2.mtime_string);//Fri Jun 12 12:04:45 PDT 2020

Date current_date = TalendDate.getCurrentDate();

if(TalendDate.diffDate(current_date, lastModifiedDate,"HH") <= 24) {

context.setProperty("file", ((String)row2.abs_path));

System.out.println(((String)globalMap.get("file"))); 

output_row.abs_path = input_row.abs_path;

System.out.println(output_row.abs_path);

}

BTW my filename is in the below format:A_B_C_D_E_20200926053617.csv

Let me know what is incorrect here?

[Haresh]

tmap goes like this:

[Haresh]

I think more like what comes after tmap,i tried using tlogrow and tftpget and it just shows "mtime_string" cannot be resolved to a variable?So how do i display or retrieve the sorted/files based on tmap thereon because i keep running into an error whatsoever !

[Anonymous]

It should be row2.mtime_string in the filter expression.@Harryishs

[Anonymous]

The component is developed in Java code. For now, there is no FTP java API can read the latest files only. We have to iterate all files.