topic Re: Exclude File Mask in TFTPFileList in Talend Studio

Exclude File Mask in TFTPFileList

Anonymous — Sat, 16 Nov 2024 08:03:14 GMT

Hello,

I am using the TFTPFileList component, but unlike the TFileList component which has exclude file mask option in advance settings, I do not see this option in the TFTPFileList component. I want to exclude all the "xml" files and files with certain substrings that I am transferring from my remote file system to Hbase. Is there a way I can do this without storing any files or scripts on my local computer through which Talend is running? I would appreciate any help asap!

Thanks!

Re: Exclude File Mask in TFTPFileList

Anonymous — Thu, 21 Jun 2018 18:31:48 GMT

By the way I have a large number of files of different formats, and so I would really like to exclude just the xml files then to include all the 100 other files with different formats.

Re: Exclude File Mask in TFTPFileList

iamabhishek — Mon, 25 Jun 2018 14:39:48 GMT

One way of achieving this would be to use Java code to filter out the files which you don't want to process in your subsequent steps.

Code for tJava -

String s = ((String)globalMap.get("tFTPFileList_1_CURRENT_FILE")).substring(StringHandling.INDEX( ((String)globalMap.get("tFTPFileList_1_CURRENT_FILE")),".")).trim();
if(!s.equals(".xml")){
System.out.println(s);
context.fileName = ((String)globalMap.get("tFTPFileList_1_CURRENT_FILE"));
globalMap.put("tFTPFileProperties_2_FILENAME", null);
}

The above could would make sure that the immediate next component of tJava (in my example (tFTPFileProperties) won't receive the any xml file as part of it's iteration.

Sample job flow -

tFTPFileProperties settings -

Re: Exclude File Mask in TFTPFileList

Anonymous — Mon, 25 Jun 2018 14:54:37 GMT

Thank You for your reply. I did figure it out by trial error. I guess I should have posted my solution earlier as well. I basically did the same as you illustrated in your response, however the only difference is: you did:

System.out.println(s);

whereas I just:

continue;

as I did not want anything to print out. However, thank you for your response as this not only just confirms my trial and error method of excluding files but can help others in the future.

Re: Exclude File Mask in TFTPFileList

Anonymous — Fri, 03 Aug 2018 15:13:25 GMT

Hi,

I'm having the same problem, of missing the Exluding Filemask in tFTPFileList, but this JavaCode is nog working for me..

I tried it exactly the same as you put it, except that my next component is a tFTPGet, but all the files go through.. it doesn't stop the ones that should be excluded.

Am I doing something wrong?

I would like some more help here, cause I'm getting desperate with my job.

Thanks in advance!

Re: Exclude File Mask in TFTPFileList

Anonymous — Sat, 04 Aug 2018 00:12:52 GMT

Hi,

Let us consider your TFTPFileList is named : "tFTPFileList_1"

So for example if I wanted to exclude all that ended with "xml", in my java component, I coded: if(((String)globalMap.get("tFTPFileList_1_CURRENT_FILEPATH")).endsWith(".xml")) {

continue;

}

or if I wanted to exclude all the files that begin with "sony", than in my java component, I coded:

if(((String)globalMap.get("tFTPFileList_1_CURRENT_FILEPATH")).startsWith("sony")) {

continue;

}

This java component will be attached right after your TFTPFileList in order to disregard the files you do not want to pass into your FTPGet.

You will have to figure out a regex in java to parse/obtain the filenames of your files like the ones above (i.e. startsWith("") or contains(""), etc to disregard the corresponding files you wish.

Next create a context variable named, say: filename

Next, connect another java component to this first java component and input the following code:

context.filename = ((String)globalMap.get("tFTPFileList_1_CURRENT_FILE"));

Now connect this to your TFTPGet component. Fill in the details of the local and remote directory of the TFTPGet Component. Finally, in your Filemask of TFTPGet component, add the context.filename as a "Files" parameter.

This should get your desired files.

Hope this helps!

Re: Exclude File Mask in TFTPFileList

Anonymous — Sat, 04 Aug 2018 00:14:22 GMT

Hi,

Let us consider your TFTPFileList is named : "tFTPFileList_1"

So for example if I wanted to exclude all that ended with "xml", in my java component, I coded: if(((String)globalMap.get("tFTPFileList_1_CURRENT_FILEPATH")).endsWith(".xml")) {

continue;

}

or if I wanted to exclude all the files that begin with "sony", than in my java component, I coded:

if(((String)globalMap.get("tFTPFileList_1_CURRENT_FILEPATH")).startsWith("sony")) {

continue;

}

This java component will be attached right after your TFTPFileList in order to disregard the files you do not want to pass into your FTPGet.

You will have to figure out a regex in java to parse/obtain the filenames of your files like the ones above (i.e. startsWith("") or contains(""), etcSmiley Wink to disregard the corresponding files you wish.

Next create a context variable named, say: filename

Next, connect another java component to this first java component and input the following code:

context.filename = ((String)globalMap.get("tFTPFileList_1_CURRENT_FILE"));

Now connect this to your TFTPGet component. Fill in the details of the local and remote directory of the TFTPGet Component. Finally, in your Filemask of TFTPGet component, add the context.filename as a "Files" parameter.

This should get your desired files.

Hope this helps!