tfilelist won't ingest more than one file?

[JSey]

Hi

I'm using a job where files are dumped into a sftp daily, and the filename is a pattern of some static text follow by date. I have a tfilelist compoent to use a mask to read the file, which a tfileinputdelimited later to call the files. I want it to read more than one file that qualifies within the mask (in case of script failure and there are multiple files from different dates), but I can only get the tfileinputdelimited to read one file. Not sure why this is happening. Little help please, thanks.

Here are the two components:

tfilelist:

tfileinputdelmited:

[Anonymous]

Hi

Can you show us an example of your file name? For debugging, you can print the value of global variable ((Integer)globalMap.get("tFileList_1_NB_FILE")) to check how many files are found.

Regards

Shong

[manodwhb]

@Johnny Sey​ ,Can you show you job design so that we can help you on the problem.

could you please re-check that do you have multiple files in source directory with dc*.csv?

Thanks,

Manohar

[JSey]

@Shicong Hong​ 

Thank you for your suggestion. I added the following line to output:

System.out.println((Integer)globalMap.get("tFileList_4_NB_FILE"));

And the output looks like this:

Starting job ********_Salesforce_v1 at 14:42 11/01/2021.

[statistics] connecting to socket on port 3353

[statistics] connected

1

2

1

2

1

2

1

2

I'm confused why it is 1 and 2? When I check on screen, I know the file has 511 lines, and Talend was only reading 511 lines.

Here is the filemask in tfilelist_4

"bk*.csv"

Here are the actual filenames in the folder:

bk.201201-bk.91.csv

bk.201201-bk.91 - Copy.csv

I was expecting 1022 lines, not 511 lines. Please help? Thanks.

[Anonymous]

the global variable (Integer)globalMap.get("tFileList_4_NB_FILE") should be used in next subjob, please share a screenshot of your job design? It will be useful for us to address the issue.

[JSey]

@Shicong Hong​ 

Thanks for your help. Here are screenshots of the first part of the script and the last. The middle are just more of the same (read a file, some tmap and push to Salesforce)

Beginning:

 

End:

Right now I have the tJava_1 after the tfilelist_4. For this example, I'm trying to read multiple BK file (tfilelist_4). In the second screenshot, the BK file is read in a tfileinputdelimited, and I think it is only reading one file because the number of rows only matches one file (I duplicated the file and made a copy).

Any suggestion would be fantastic. Basically I want it to read and process all files that fits the mask in tfilelist_4 (bk*.csv).

[Anonymous]

You don't use the tFileList component correctly, this component is used to iterate the files one by one in the specified folder, for example (read BK file):

tFileList_4--iterate--tFileInputDelimited-->.main->tLogRow

on the FileName/stream field of tFileInputDelimited, set the file path as

((String)globalMap.get("tFileList_4_CURRENT_FILEPATH"))

If there are total 2 files are matched the file marks, tFileInputDelimited will execute 2 times to read each file.

[JSey]

@Shicong Hong​ 

Thank you for your help. In the tfileinputdelmited, this is what I had:

((String)globalMap.get("tFileList_4_CURRENT_FILEPATH"))

But I am don't have a tlogrow component. I haven't use tlogrow before. So it goes after the tfileinputdelimited, but before the tmaps in each subjobs? Please confirm, thanks.

[JSey · Accepted solution]

Ok, I found the issue. The way I had the jobs setup, the tfilelist was in one subjob, and the tfileinputdelimited was in another subjob. This was the problem and once I moved the tfilelist to be in the same subjob as the tfileinputdelimited, it works fine. Thanks for everyone's help.

[Anonymous]

Great, thanks for your feedback!

Regards

Shong