Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.
Hi
I'm using a job where files are dumped into a sftp daily, and the filename is a pattern of some static text follow by date. I have a tfilelist compoent to use a mask to read the file, which a tfileinputdelimited later to call the files. I want it to read more than one file that qualifies within the mask (in case of script failure and there are multiple files from different dates), but I can only get the tfileinputdelimited to read one file. Not sure why this is happening. Little help please, thanks.
Here are the two components:
tfilelist:
tfileinputdelmited:
Ok, I found the issue. The way I had the jobs setup, the tfilelist was in one subjob, and the tfileinputdelimited was in another subjob. This was the problem and once I moved the tfilelist to be in the same subjob as the tfileinputdelimited, it works fine. Thanks for everyone's help.
Hi
Can you show us an example of your file name? For debugging, you can print the value of global variable ((Integer)globalMap.get("tFileList_1_NB_FILE")) to check how many files are found.
Regards
Shong
@Johnny Sey ,Can you show you job design so that we can help you on the problem.
could you please re-check that do you have multiple files in source directory with dc*.csv?
Thanks,
Manohar
@Shicong Hong
Thank you for your suggestion. I added the following line to output:
System.out.println((Integer)globalMap.get("tFileList_4_NB_FILE"));
And the output looks like this:
Starting job ********_Salesforce_v1 at 14:42 11/01/2021.
[statistics] connecting to socket on port 3353
[statistics] connected
1
2
1
2
1
2
1
2
I'm confused why it is 1 and 2? When I check on screen, I know the file has 511 lines, and Talend was only reading 511 lines.
Here is the filemask in tfilelist_4
"bk*.csv"
Here are the actual filenames in the folder:
bk.201201-bk.91.csv
bk.201201-bk.91 - Copy.csv
I was expecting 1022 lines, not 511 lines. Please help? Thanks.
the global variable (Integer)globalMap.get("tFileList_4_NB_FILE") should be used in next subjob, please share a screenshot of your job design? It will be useful for us to address the issue.
@Shicong Hong
Thanks for your help. Here are screenshots of the first part of the script and the last. The middle are just more of the same (read a file, some tmap and push to Salesforce)
Beginning:
End:
Right now I have the tJava_1 after the tfilelist_4. For this example, I'm trying to read multiple BK file (tfilelist_4). In the second screenshot, the BK file is read in a tfileinputdelimited, and I think it is only reading one file because the number of rows only matches one file (I duplicated the file and made a copy).
Any suggestion would be fantastic. Basically I want it to read and process all files that fits the mask in tfilelist_4 (bk*.csv).
You don't use the tFileList component correctly, this component is used to iterate the files one by one in the specified folder, for example (read BK file):
tFileList_4--iterate--tFileInputDelimited-->.main->tLogRow
on the FileName/stream field of tFileInputDelimited, set the file path as
((String)globalMap.get("tFileList_4_CURRENT_FILEPATH"))
If there are total 2 files are matched the file marks, tFileInputDelimited will execute 2 times to read each file.
@Shicong Hong
Thank you for your help. In the tfileinputdelmited, this is what I had:
((String)globalMap.get("tFileList_4_CURRENT_FILEPATH"))
But I am don't have a tlogrow component. I haven't use tlogrow before. So it goes after the tfileinputdelimited, but before the tmaps in each subjobs? Please confirm, thanks.
Ok, I found the issue. The way I had the jobs setup, the tfilelist was in one subjob, and the tfileinputdelimited was in another subjob. This was the problem and once I moved the tfilelist to be in the same subjob as the tfileinputdelimited, it works fine. Thanks for everyone's help.
Great, thanks for your feedback!
Regards
Shong