topic Re: [resolved] Not able to process multiple files using tFileList in Talend Studio

[resolved] Not able to process multiple files using tFileList

Anonymous — Thu, 10 Jul 2014 22:31:52 GMT

I am unable to configure Talend to process multiple files. I have a job that transforms TXT files to HL7 files and that part of the job is working.
I am trying to iterate on a folder with multiple TXT files to transform them to HL7. I am using tFileList as the component to provide this function. There are 10 TXT files to be process in the source directory.
When I run the job, only the last of the 10 txt files is transformed.
Please help!!!

Re: [resolved] Not able to process multiple files using tFileList

Anonymous — Fri, 11 Jul 2014 03:49:06 GMT

Hi,
How did you set your tHl7output component? Could you please try to use tFileOutPutDelimited with checked "Append" option instead of tHl7output component to see if your job works well?
Best regards
Sabrina

Re: [resolved] Not able to process multiple files using tFileList

Anonymous — Fri, 11 Jul 2014 03:53:34 GMT

Hi xdshi,
thank you for your response. OK, I'll replace the HL7 Output with the fileoutputdelimited control. Once thing I noticed is the connector to the HL7 Output is a main'merge' whereas all examples I've seen do not have a 'merger' connnector. I'll try shortly and post my results... thanks again...
Please note that I am happy to post the job up somewhere if you or anyone from talend would like to have a look...

Re: [resolved] Not able to process multiple files using tFileList

Anonymous — Fri, 11 Jul 2014 04:19:26 GMT

Hi,
From your screenshots, you have already got 10 txt files by tFileList. There is an iterate link that loop on files contained in a directory, on rows contained in a file or on DB entries between tfilelist and tfileinputdelimited, that's the reason why there is 1 row in main row. The job design is correct. If you want to get all literared files, you should "Append" them.
Best regards
Sabrina

Re: [resolved] Not able to process multiple files using tFileList

Anonymous — Fri, 11 Jul 2014 05:01:27 GMT

Hi xdshi,
thanks for your response. OK. Yes, when I replaced the HL7 Output component and replaced with a tFileInputDelimited component, all 10 txt files were iterated through and 10 new txt files were created in a separate folder I defined.
It seems like the HL7 output component requires a different iteration process. As I mentioned, when using the HL7 Output component the 'Main' connector becomes a 'Main''Merge connector.
So, how would I transform multiple txt files to hl7 files? It seems like there is a different method I should be using but all instruction indicates doing it just as I have done?
Please advise and thanks again!

Re: [resolved] Not able to process multiple files using tFileList

Anonymous — Fri, 11 Jul 2014 07:12:47 GMT

Hi Sabrina, just noticed I wasn't using your correct name. sorry about that.
On a hunch a split the 3 segments of the HL7 (MSH, EVN, PID) to 3 tmap inputs. I was hoping that was the problem but unfortunately the results are the same.
Please let me know what other information I can provide. I can post the job as well if you'd like to have a look at it.
Thank you

Re: [resolved] Not able to process multiple files using tFileList

Anonymous — Fri, 11 Jul 2014 08:21:43 GMT

Hi,
Could you try to put your multiple txt files on local disk first then loading the flat file into tHL7output?
The workflow should be: tfilelist-->iterate-->tfileinputdelimited-->tfileoutputdelimited(Append)
|
onsubjobOK
|
tfileinputdelimited_1(same path with output file)-->tHL7output.
Let me know if it is OK with you.
Best regards
Sabrina

Re: [resolved] Not able to process multiple files using tFileList

Anonymous — Fri, 11 Jul 2014 10:03:26 GMT

Hi Sabrina,
I made the changes to the job as requested but the results are the same. I've posted an image of the job. The image is called "OnSubJobOK_Example".
I set the 'Append' bit as well. I've also tried both UTF-8 and ISO-8859-15 as encoding types but that did not do anything different. I don't know if that has anything to do with it but was the one thing I've not changed.

Please let me know if I can post the job somewhere for you to have a closer look at it.
I'm desperate!!!
Thanks!

Re: [resolved] Not able to process multiple files using tFileList

Anonymous — Fri, 11 Jul 2014 10:51:10 GMT

Hi,
Could you please send an email to me with your exported job? We will make a further investigation on it.
Best regards
Sabrina

Re: [resolved] Not able to process multiple files using tFileList

Anonymous — Fri, 11 Jul 2014 11:54:22 GMT

Hi Sabrina, please let me know what email address I can send an attachment to.
Thanks!

Re: [resolved] Not able to process multiple files using tFileList

Anonymous — Fri, 11 Jul 2014 13:20:51 GMT

Hi Sabrina, please let me know what email address I can send an attachment to.
Thanks!

Re: [resolved] Not able to process multiple files using tFileList

Anonymous — Fri, 11 Jul 2014 13:56:38 GMT

hi,
do a simple test.
iterate the repository with tFilelist and link to a tRunJob.
As parameter past the value of the iterate file.
in child job calling by tRunJob, put a tJava & use println() method to see each file going through the process.
when it's ok, in child job manage your data coming from file iteratevely.
if result have to go in a single file , use append.
hope it help
regards
laurent

Re: [resolved] Not able to process multiple files using tFileList

Anonymous — Fri, 11 Jul 2014 18:48:53 GMT

Hi Sabrina, I'm going to try to do what kzone recommended though at first glance I'm not sure I understand.
Would you be able to provide me with an email that I can attach the zipped job?
Thanks

Re: [resolved] Not able to process multiple files using tFileList

Anonymous — Fri, 11 Jul 2014 19:13:20 GMT

Hi kzone, thank you for your help. I am unable to understand how to 'iterate the repository with tfilelist'...
would you be able to provide additional information?
Thank you,
Ray

Re: [resolved] Not able to process multiple files using tFileList

Anonymous — Sat, 12 Jul 2014 04:35:15 GMT

Hi Sabrina, I am going to try to connect a tRunJob that will run the job that does successfully create the last of the 10 HL7 files from the TXT files. I think if I add a tFileDelete after the HL7 output, i can delete the last txt file used. I can then continue to run the run component until no TXT files are left. That will work right? Well I'm off to try it... Please keep me informed with what you are finding with the issue. Thanks!!

Re: [resolved] Not able to process multiple files using tFileList

soujanyam — Mon, 14 Jul 2014 06:21:47 GMT

hi,
I'm very new to the Talend
I too had the same problem.I'm trying to merge the multiple files into a single file which have the same schema.For that purpose I used workflow as
tFilelist-->tFileInputDelimited-->tFileOutputDelimited(append)
I used tFileInpuDelimited from Repository and in order to merge changed the property from built in into repository and changed the file path as per tFilelist global map variable. Please let me know where I did mistake.
Thanks in advance.

Re: [resolved] Not able to process multiple files using tFileList

Anonymous — Mon, 14 Jul 2014 06:43:48 GMT

Hi,
With this job design, it will delete the current txt file that is passed by tRunJob from the main job to child job.
Could you please send your current job to me( the email address is just under my avatar).
Best regards
Sabrina

Re: [resolved] Not able to process multiple files using tFileList

soujanyam — Mon, 14 Jul 2014 12:44:10 GMT

Hi Sabrina,
Could u please provide the solution to my problem

Re: [resolved] Not able to process multiple files using tFileList

Anonymous — Tue, 15 Jul 2014 08:35:41 GMT

here is an example to iterate on File repository and manage data from each file in a child jobh (just a tJava for the test) @screenshoots
regards
laurent

Re: [resolved] Not able to process multiple files using tFileList

Anonymous — Wed, 16 Jul 2014 04:44:25 GMT

Thanks Kzone, going to try this now...