Do not input private or sensitive data. View Qlik Privacy & Cookie Policy.
Skip to main content

Announcements
Join us in Toronto Sept 9th for Qlik's AI Reality Tour! Register Now
cancel
Showing results for 
Search instead for 
Did you mean: 
JSey
Creator
Creator

tfilelist won't ingest more than one file?

Hi

I'm using a job where files are dumped into a sftp daily, and the filename is a pattern of some static text follow by date. I have a tfilelist compoent to use a mask to read the file, which a tfileinputdelimited later to call the files. I want it to read more than one file that qualifies within the mask (in case of script failure and there are multiple files from different dates), but I can only get the tfileinputdelimited to read one file. Not sure why this is happening. Little help please, thanks.

Here are the two components:

tfilelist:

0693p00000AbrIdAAJ.png

tfileinputdelmited:

0693p00000AbrIsAAJ.png

Labels (2)
1 Solution

Accepted Solutions
JSey
Creator
Creator
Author

Ok, I found the issue. The way I had the jobs setup, the tfilelist was in one subjob, and the tfileinputdelimited was in another subjob. This was the problem and once I moved the tfilelist to be in the same subjob as the tfileinputdelimited, it works fine. Thanks for everyone's help.

View solution in original post

9 Replies
Anonymous
Not applicable

Hi

Can you show us an example of your file name? For debugging, you can print the value of global variable ((Integer)globalMap.get("tFileList_1_NB_FILE")) to check how many files are found.

 

Regards

Shong

manodwhb
Champion II
Champion II

@Johnny Sey​ ,Can you show you job design so that we can help you on the problem.

 

could you please re-check that do you have multiple files in source directory with dc*.csv?

 

Thanks,

Manohar

JSey
Creator
Creator
Author

@Shicong Hong​ 

 

Thank you for your suggestion. I added the following line to output:

 

System.out.println((Integer)globalMap.get("tFileList_4_NB_FILE"));

 

And the output looks like this:

 

Starting job ********_Salesforce_v1 at 14:42 11/01/2021.

[statistics] connecting to socket on port 3353

[statistics] connected

1

2

1

2

1

2

1

2

 

I'm confused why it is 1 and 2? When I check on screen, I know the file has 511 lines, and Talend was only reading 511 lines.

 

Here is the filemask in tfilelist_4

 

"bk*.csv"

 

Here are the actual filenames in the folder:

 

bk.201201-bk.91.csv

bk.201201-bk.91 - Copy.csv

 

I was expecting 1022 lines, not 511 lines. Please help? Thanks.

Anonymous
Not applicable

the global variable (Integer)globalMap.get("tFileList_4_NB_FILE") should be used in next subjob, please share a screenshot of your job design? It will be useful for us to address the issue.

JSey
Creator
Creator
Author

@Shicong Hong​ 

 

Thanks for your help. Here are screenshots of the first part of the script and the last. The middle are just more of the same (read a file, some tmap and push to Salesforce)

 

Beginning:

0693p00000BBceWAAT.png 

End:

0693p00000BBcegAAD.pngRight now I have the tJava_1 after the tfilelist_4. For this example, I'm trying to read multiple BK file (tfilelist_4). In the second screenshot, the BK file is read in a tfileinputdelimited, and I think it is only reading one file because the number of rows only matches one file (I duplicated the file and made a copy).

 

Any suggestion would be fantastic. Basically I want it to read and process all files that fits the mask in tfilelist_4 (bk*.csv).

Anonymous
Not applicable

You don't use the tFileList component correctly, this component is used to iterate the files one by one in the specified folder, for example (read BK file):

tFileList_4--iterate--tFileInputDelimited-->.main->tLogRow

 

on the FileName/stream field of tFileInputDelimited, set the file path as

((String)globalMap.get("tFileList_4_CURRENT_FILEPATH"))

 

If there are total 2 files are matched the file marks, tFileInputDelimited will execute 2 times to read each file.

 

 

 

 

 

 

 

 

JSey
Creator
Creator
Author

@Shicong Hong​ 

 

Thank you for your help. In the tfileinputdelmited, this is what I had:

 

((String)globalMap.get("tFileList_4_CURRENT_FILEPATH"))

 

But I am don't have a tlogrow component. I haven't use tlogrow before. So it goes after the tfileinputdelimited, but before the tmaps in each subjobs? Please confirm, thanks.

JSey
Creator
Creator
Author

Ok, I found the issue. The way I had the jobs setup, the tfilelist was in one subjob, and the tfileinputdelimited was in another subjob. This was the problem and once I moved the tfilelist to be in the same subjob as the tfileinputdelimited, it works fine. Thanks for everyone's help.

Anonymous
Not applicable

Great, thanks for your feedback!

 

Regards

Shong