Is there a way in talend to find the files from the last 24 hours or within 1 day without reading through entire file list from ftp

[Haresh]

The ftp folder has around 4000-5k files present there and i dont want to be passing them into a bufferinput/output component and after that sort them using tmap function .Instead i would prefer without bothering the no.of files in the ftp folder directly can i find the files from last day(24 hours timeframe) and use ftpget?

I dont want to go through the hassle of storing/reading all the files in a bufferedreader and sort/arrange by mtime_desc or by other ftpfileproperties method.Instead a direct way to fetch and retrieve the most recent files from previous day would be quite faster and effective?

[Anonymous]

Hi

Due to API limitation, no a direct way to get only the latest files from FTP server. There has been some similar discussions on the community, please read

https://community.talend.com/s/feed/0D53p000087f6PHCAY

Regards

Shong

[JohnRMK]

Hello,

I had a use case recently with the same problem but it was purging files according to a filter

What I suggest to do is use a shell or powershell with the tSystem command to retrieve the files of interest and then retrieve them via tFTPGet.

Get-ChildItem -Path $env:ProgramFiles -Recurse -Include *.exe | Where-Object -FilterScript {($_.LastWriteTime -gt '2005-10-01') -and ($_.Length -ge 1mb) -and ($_.Length -le 10mb)}

https://docs.microsoft.com/fr-fr/powershell/scripting/samples/working-with-files-and-folders?view=powershell-7.1

[Haresh]

Thanks for your explanation,i was looking for something along these lines and it would be even helpful if you could add a snapshot of the job flow how it looks like so i can have a better understanding.I assume that this would need to go into the powershell command:

Get-ChildItem -Path $env:ProgramFiles -Recurse -Include *.exe | Where-Object -FilterScript {($_.LastWriteTime -gt '2005-10-01') -and ($_.Length -ge 1mb) -and ($_.Length -le 10mb)}

and in the case wherein i would need to retrieve the files from current day (sysdate) -1 (previous day) history alone i have to change the lastwritetime parameter i believe.

Let me try this out and get back to you,coz for me reading through all the files in the ftp folder and sorting them later based on tmap to find files from 24 hours takes a huge amount of processing time.

So i would prefer a quicker and effective approach whichever fits better