Parallel execution takes more time than the non-parallel execution

[Anonymous]

Hi Team,

Followed the instructions described in the below link to develop a job execution in a loop:

https://help.talend.com/reader/mjoDghHoMPI0yuyZ83a13Q/iL2h45sTpz~InS1_0iOj5w

if I disable the component tSleep and select the check box for Parallel Execution, job takes more time to complete than the non parallel execution.

Attached the document that has all the details about the job design and execution results.

Can you please let me how to create a job with tloop(iterate) enabling  parallel execution so that .parallel execution  takes less time than the non-parallel execution.

Thanks.

[Anonymous]

Hi Nikhil,

Apologize for the typo and here is the edited issue:

Issue - Parallel execution(with number of parallel execution 2) took 338 milli seconds where as serial execution took 49 milli seconds to get the document name of 15 PDF files(just file name without reading file content) in a directory.

Could you please let me know why  parallel execution is taking more time than the serial execution.

Thanks for your response.

[Anonymous]

Hi,

    The first verification I will do will be to increase the memory parameters and see the performance results.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

[Anonymous]

Hi Nikhil,

Increased the memory for main job as well as sub job as below:

-Xms1024M

-Xmx1792M

Parallel execution took 338 milli seconds where as serial execution took 46 milli seconds.

Thanks.

[Anonymous]

Hi,

     You will see more difference in the performance for a longer job execution processes. 

     Since your job is executing in milli seconds, I would pick either of them as end of day, its a batch process.

     My view is that since you are using Open source version of Talend, the number of options in front of you will be limited. If you are using a mission critical application with lot of bench marking required, I would opt the enterprise version of Talend, which is having more parallel processing features.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

[Anonymous]

Hi Nikhil,

As I mentioned in my previous post, am using the 30 days trail version of Talend Data Integration 6.5.1 to evaluate enabling parallel execution feature for iterate loop. 

Downloaded the trail version of Talend Data Integration 6.5.1 from the below link:

https://www.talend.com/download/

Can you please provide a working sample job design (if you have one) for enabling parallel execution for iterate loop. 

Thanks,

Narayana.

[Anonymous]

Hi,

    I would recommend you to go through the tParallelize component along with sample scenarios.

https://help.talend.com/reader/hCrOzogIwKfuR3mPf~LydA/J2Gx1RZDoO6xyhQalZJqUQ

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

[Anonymous]

Hi Nikhil,

Thanks for the response. Will explore the tParallelize component and update you.

Do you see any issue with enabling Parallel execution for Iterate loop or do not suggest to use it?

Thanks,

Narayana.