topic Re: Get two latest files from ftp directory in Talend Studio

Get two latest files from ftp directory

RA6 — Sat, 16 Nov 2024 07:23:39 GMT

Hello folks,

I am trying to retrieve two last files below [different names] from an ftp directory.

Expected result :

Lastest File 1 : Clients_26102018.csv

Lastest File 2 : Ecritures_26102015.csv

Based on the tftpfileproperties, i have filtered the mtime by desc and fetching the first row and i am retrieving my first file => Clients_26102018.csv

As you can see, i am using filemask "Client_*.csv" to retrieve the file.

Apart from duplicating the same job for the other file, I want to know if there is another way to achieve this.

something like foreach parameters? The only thing thing that is changing is the file mask.

Or is there way to retrieve both files in one job?

Thank you.

Best regards,

Ashish

Re: Get two latest files from ftp directory

Anonymous — Mon, 29 Oct 2018 14:33:01 GMT

Hi Ashish,

Why don't you fetch all the file property details to buffer by giving the file mask as "*"? Once you get the file details, you can group the files to different groups based on file names (you will need some regex/substring function here based on your use case) and pick the file with oldest date.

Once the required file is identified in each group, you can do file fetch for only those files.

Warm Regards,

Nikhil Thampi

Re: Get two latest files from ftp directory

RA6 — Tue, 30 Oct 2018 15:02:20 GMT

@nthampi

Thank you for your response.
I am new to talend. Can you show me how to group the files to different groups based on file names please?

Best regards,
Asadasing

Re: Get two latest files from ftp directory

Anonymous — Fri, 02 Nov 2018 10:57:34 GMT

Hi,

First of all my sincere apologies for responding back. I am currently on vacation but thought to update this post as per my promise.

I have created the current flow using tFIlelist in local but you can create the same logic for FTP flow also using tFTP components.

Below are the sample files I have created and my intention is to pick the latest files(highlighted) based on file creation timestamp.

The job flow below is picking the files and the last subjob can be used (instead of tfileproperties_2, you can use other components) to process them further.

I am attaching the job for your reference. It is quite self explanatory. The only change you need to do is to use the FTP components for file list and reading.

If the answer has helped you, could you please mark the topic as resolved? Kudos are also welcome 🙂

Warm Regards,

Nikhil Thampi

child.zip

Re: Get two latest files from ftp directory

RA6 — Fri, 02 Nov 2018 11:26:33 GMT

Genius!!
Thank you @nthampi

Best regards,
asadasing

Re: Get two latest files from ftp directory

Anonymous — Sat, 03 Nov 2018 07:11:30 GMT

my version is 6.5.1