topic Re: How to get latest files in Talend Studio

How to get latest files

Anonymous — Tue, 06 Feb 2018 11:42:25 GMT

How to get the latest file. when we run history. Can anyone please help to sort it out.

Re: How to get latest files

TRF — Tue, 06 Feb 2018 12:16:11 GMT

Have a look to tFileList component options to sort files based on chronological order with older first.

Re: How to get latest files

Anonymous — Tue, 06 Feb 2018 18:39:29 GMT

There is only modified in tfilelist.

In my scenario I'm downloading all the history files in job server, but based on filedate it should process only latest date file.

Eg: 1.Customersdata_20180126.psv

2.Customersdata_20180201.psv

3.Customersdata_20171231.psv

We have to process only 2.Customersdata_20180201.psv file which is latest

Thanks,

Venkat

Re: How to get latest files

TRF — Tue, 06 Feb 2018 20:13:27 GMT

Same answer based on filename.
Memorize the 1rst name using a global variable then reuse the variable in the next subjob to get the file.

Re: How to get latest files

Anonymous — Wed, 07 Feb 2018 05:29:47 GMT

Can you please explain in detail, because file names may vary.

Re: How to get latest files

manodwhb — Wed, 07 Feb 2018 05:59:18 GMT

first put in the memory that the pattern of the file. and call the same pattern in to tFilelist.

1) pattern would be like this globalMap.put("pattern", "Customersdata_"+ TalendDate.getDate("yyyyMMdd") + pattern_suffixe);

2) use the same paatern to get see the below screen shot.

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.

Re: How to get latest files

TRF — Wed, 07 Feb 2018 09:35:27 GMT

Hi,

Stricly based on your filenames, here is a complete solution to get the latest file:

tFileList is configured with the filemask "*.psv" and here is the content of the tJava component:

// get the date contained into the current filename
int dateFile = Integer.parseInt(((String)globalMap.get("tFileList_2_CURRENT_FILE")).replaceAll("^.*_", "").replace(".psv", ""));

// if there is no "maxDate" known or the current date is > to "maxDate" then the current date becomes the "maxDate"
if ((Integer)globalMap.get("maxDate") == null || dateFile > (Integer)globalMap.get("maxDate")) {
   globalMap.put("maxDate", dateFile);
   globalMap.put("lastFile", (String)globalMap.get("tFileList_2_CURRENT_FILE"));
}

Now you can get the filename using the global variable called "lastFile" using the following syntax (for example to populate the Filename field of a tFileInputDelimited or tFileInputFullRow component):

(String)globalMap.get("lastFile")

This example is based on your input and works fine.

You may have to adapt it to your case.

Don't forget to mark your case as solved as soon as it is ok for you (Kudo also accepted).

Re: How to get latest files

Anonymous — Wed, 07 Feb 2018 15:23:39 GMT

Hi,

I'm very thankful for your detailed explanation.

My filename is coming like this

Good_Customersdata_20171231_20171231092635.psv

In this file 20171231092635 is date with timestamp I can to check based on this.

Please help me to sort it

Re: How to get latest files

TRF — Wed, 07 Feb 2018 17:36:55 GMT

Just have to change the tJava code to use BigInteger instea of Int:

BigInteger dateFile = new BigInteger(((String)globalMap.get("tFileList_2_CURRENT_FILE")).replaceAll("^.*_", "").replace(".psv", ""));

if ((BigInteger)globalMap.get("maxDate") == null || dateFile.compareTo((BigInteger)globalMap.get("maxDate")) == 1) {
	globalMap.put("maxDate", dateFile);
	globalMap.put("lastFile", (String)globalMap.get("tFileList_2_CURRENT_FILE"));
}

Re: How to get latest files

Anonymous — Wed, 07 Feb 2018 18:24:22 GMT

It is showing BigInt cannot be resolved.

Re: How to get latest files

TRF — Wed, 07 Feb 2018 19:25:53 GMT

Add the following in the tJava component Advanced settings:

import java.math.*;

Re: How to get latest files

manodwhb — Thu, 08 Feb 2018 05:48:13 GMT

did you try my solution?

Re: How to get latest files

Anonymous — Thu, 08 Feb 2018 05:52:53 GMT

Thanks for your support

Actually I was confused, I tried but didn't understand

Re: How to get latest files

manodwhb — Thu, 08 Feb 2018 06:01:50 GMT

1) in tjava put below code.

String pattern_suffixe = "*.csv";

globalMap.put("yyyyMMdd", TalendDate.getDate("yyyy_MM_dd"));

globalMap.put("pattern", "Customersdata_"+ TalendDate.getDate("yyyyMMdd") + pattern_suffixe);

2) in tFilelist use the below parameter under the filemask.same which i have provided in screenshot in the first reply.

((String)globalMap.get("pattern"))

Re: How to get latest files

manodwhb — Thu, 08 Feb 2018 06:44:28 GMT

is my solution usefull?

Re: How to get latest files

TRF — Thu, 08 Feb 2018 07:19:47 GMT

Finally you will be able to get the file with using the variable "lastFile" as a filename:

Now you got a complete solution for your case.

Don't forget to mark it as solved (Kudos also accepted).

Re: How to get latest files

Anonymous — Sun, 11 Feb 2018 09:37:09 GMT

Hi ,

Thanks for your support, I will accept your solution.

I found one simple way to get the latest file.

In tfilelist we have by filename instead of default and order by des.

Then we will get latest file.

Thanks