topic Re: [resolved] Concurrent Subjob execution. in Talend Studio

[resolved] Concurrent Subjob execution.

sammierlening — Mon, 14 Jul 2014 15:41:00 GMT

Is it possible to call a Subjob over a iterate connection and not wait for sequential execution completion?
Lets say I have a folder with 10 files. I would like to process 4 files concurrently, passing each filename to a subjob for processing. Currently, 1 file name is passed to the subjob, when that subjob is complete, the next filename is passed. What I would like if for each filename to be passed to a subjob, the subjobs to execute concurrently, and once all 4 are finished, the next batch of four filenames are passed to subjobs.

Re: [resolved] Concurrent Subjob execution.

Anonymous — Mon, 14 Jul 2014 16:17:08 GMT

I would recommend you to have four subjobs which are running parallely as independent process in main job...
Inside your subjobs, once the file is processed, move to archive folder and it would do what you are expecting..
vaibhav

Re: [resolved] Concurrent Subjob execution.

sammierlening — Mon, 14 Jul 2014 16:59:55 GMT

Solved.

Re: [resolved] Concurrent Subjob execution.

Anonymous — Tue, 15 Jul 2014 08:17:57 GMT

Hi,
If the issue is closed, could you please click the "Set this topic as resolved" link which is right underneath your initial post? This way, other users will be informed that this thread has been resolved.
Many thanks
Best regards
Sabrina

Re: [resolved] Concurrent Subjob execution.

Anonymous — Tue, 15 Jul 2014 08:56:22 GMT

Hi,
Can you please provide a screenshot or approach used to solve the current problem.
thanks
vaibhav

Re: [resolved] Concurrent Subjob execution.

sammierlening — Tue, 15 Jul 2014 18:27:29 GMT

sanvaibhav, I use a tjava component coming out of the first tFileList iterate to determine which batch a file should process in. I map that value to the globalMap. Then I use a tLoop to loop each batch from 1 to the max batch number in the globalMap. Finally, I use a second tFileList to iterate each file in each batch. I changed the iterate link to "Enable parallel execution". Each filepath is passed to a subjob for each file in each batch.

Re: [resolved] Concurrent Subjob execution.

Anonymous — Wed, 16 Jul 2014 08:46:59 GMT

Oh.. that's great...
Thanks for the information and update.
vaibhav