topic Re: parallelization in Talend Studio

parallelization

Anonymous — Tue, 05 Mar 2019 16:39:45 GMT

Hello All, I have a question

I have nearly 30 jobs , which i need to run daily, so i wanted to do parellization for those 30 jobs , which way do you think it is better. Can anyone please suggest a solution for me.

Thanks In Advance

Manish

Re: parallelization

DataTeam1 — Tue, 05 Mar 2019 17:03:23 GMT

You can build jobs as stand alone (https://help.talend.com/reader/JdTBzKszzXoWvjpEJD3EBA/V5yP4nItw687WwHLVpxwFQ) and setup the execution plan for example in Windows scheduler or cron on server.

They can be executed at the same time if they are independent.

Re: parallelization

Anonymous — Tue, 05 Mar 2019 17:10:15 GMT

So, i should select all 30 jobs and from the tree view and make them into standalone and then process it as per documentation in the link

Re: parallelization

DataTeam1 — Tue, 05 Mar 2019 17:21:23 GMT

No, you should build separately each job as stand alone. If you have job which runs other subjob you have to right clik only on the main job in repository view and select build job.

Re: parallelization

DataTeam1 — Tue, 05 Mar 2019 23:13:42 GMT

If you want to run many subjob in parallel in main job read this: https://community.talend.com/t5/Design-and-Development/Can-I-run-different-subjobs-in-parallel-in-a-job/ta-p/21648

Re: parallelization

Anonymous — Wed, 06 Mar 2019 06:04:48 GMT

@ksingh

Could you please refer the Talend help link below? It is having multiple options. Please note that some of the options are available only for subscription product.

Based on the product you are using, could you please select appropriate strategy?

https://help.talend.com/reader/MNcEDgjyM49yQ58GboyG4Q/efgfWM4DK8gYYWPhJCnTjw

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved 🙂

Re: parallelization

Anonymous — Wed, 06 Mar 2019 14:44:14 GMT

@nthampi the link you sent me

https://help.talend.com/reader/MNcEDgjyM49yQ58GboyG4Q/efgfWM4DK8gYYWPhJCnTjw

I just followed same steps i need to execute 30 jobs, initially tried with 4 jobs, this worked accurately as per my requirement , thanks @nthampi

Re: parallelization

Anonymous — Wed, 06 Mar 2019 15:26:45 GMT

Hi,

Please note that you may not be able to run 30 jobs in parallel unless your job server is having sufficient compute capacity. Each subjob you are calling will consume some CPU and memory. So the wrapper job's Xms and Xmx parameters should be high enough to cater the resource requirements of subjobs.

So tparallelize means you can do parallel processing as long as you are providing sufficient compute resources.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved 🙂