topic Re: Execute multiple flows in parallel after one component in Talend Studio

Execute multiple flows in parallel after one component

Anonymous — Wed, 30 Jan 2019 16:58:47 GMT

Hi, I am trying to create a job with TOS DI 7.0.1, where I can have multiple flows run in parallel after one component say for example a tJava component. Below is a screen shot. In the first step I want to create a concatenated string which will be the parameter for tDBInput_2,tDBInput_3,tDBInput_4,tDBInput_5 and tDBInput_6. Once that is created i am doing little bit more manipulation inside tJava_2. Once that is done i want to execute tDBIput_2,tDBIput_3,tDBIput_4,tDBIput_5 and tDBIput_6 all in parallel. How can i achieve that. I do not see tParallize in pallet. Thanks in advance.

Re: Execute multiple flows in parallel after one component

David_Beaty — Wed, 30 Jan 2019 17:11:08 GMT

Hi,

That component is probably missing because you might be using the Open Source version of the product (which doesnt have that functionality).

You can get around this by creating a subjob with just the 5 tDBINput flows in, then link that subjob in to your current job. Ensure on the subjob, properties enable multi-thread execution.

The only caveat with any parallel processing strategy is consider whether your Talend execution server or database server can cope with these 5 processes being run together.

Re: Execute multiple flows in parallel after one component

Anonymous — Wed, 30 Jan 2019 17:17:02 GMT

Thanks for the response. Is there any way i can get around this without a sub job?

Re: Execute multiple flows in parallel after one component

akumar2301 — Wed, 30 Jan 2019 19:03:01 GMT

There could be many ways to implement 
 
Step 1 ) use Multi-Thread execution Option in TOS
Step 2)
option1) tWaitforFile -
 
At the end of your main subjob , create/update/delete a file which will trigger all your subjob in parralel using tWaitforFile ( go with exit loop option) . Below link will help you how it work.
 
https://help.talend.com/reader/wDRBNUuxk629sNcI0dNYaA/bnMDJWu5G0EaJl8_KxcYyA
 
Main Subjob -- > trigger a action
twaitforfile_1 --> (main) --> tlogrow --> on subjobOK --> tdbinput_1
twaitforfile_2 --> (main) --> tlogrow --> on subjobok --> tdbinput_2
 
option2) tLoop ( While condition) 
 
Create a global var VAR1 with which is true. At the end of your main subjob make VAR1 as false.
 
tLoop_1--> while VAR1  is true --> tJava ( does nothing)
|
On Subjob Ok ( when VAR1 will become false) 
[
tDbInput_1
 
Both above approach error handling of main subjob should be done properly.

Option3) Use multiple tPostJob ( if not already used and Not recommended option)
 
tPostJob_1 --> tDbinput_1
tPostJob_2 --> tDbinput_2
 
There could many other ways .
Let me know if it works.
 
 