topic Re: Parallel execution takes more time than the non-parallel execution in Talend Studio

Parallel execution takes more time than the non-parallel execution

Anonymous — Mon, 30 Sep 2019 19:15:27 GMT

Hi Team,

Followed the instructions described in the below link to develop a job execution in a loop:

https://help.talend.com/reader/mjoDghHoMPI0yuyZ83a13Q/iL2h45sTpz~InS1_0iOj5w

if I disable the component tSleep and select the check box for Parallel Execution, job takes more time to complete than the non parallel execution.

Attached the document that has all the details about the job design and execution results.

Can you please let me how to create a job with tloop(iterate) enabling parallel execution so that .parallel execution takes less time than the non-parallel execution.

Thanks.

Re: Parallel execution takes more time than the non-parallel execution

Anonymous — Tue, 01 Oct 2019 04:29:39 GMT

Hi,

The parallel execution depends on lot of parameters like number of available threads, memory availability for the Talend job, CPU utilization etc.

Could you please specify your use case for parallel execution? I would recommend to start using parallel processing in Talend using the component tParallelize. Please also increase your Xms and Xmx parameters of the job. You will see the difference in performance when you are running a job with longer processing time.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

Re: Parallel execution takes more time than the non-parallel execution

Anonymous — Tue, 01 Oct 2019 10:43:05 GMT

Hi Nikhil,

Thanks for the response.

My use case is to process the thousands of files present in the main directory as well as sub directories.

Once we read the document present in each main dir as well as sub dir, will send it to Apache Solr API for indexing.

To improve the performance, thought of using the Parallel Execution.

tParallelize component is not available by default in TOS-7.1.1

Is there any known issue with using option Parallel Execution?

My system configuration is 16 GB RAM and i7-4790 processor.

Attachment has all the details about job design and execution results.

Kindly let me know your thoughts on how we can use parallel execution for the above use case to improve performance.

Thanks.

File Processing Execution Results.docx

Re: Parallel execution takes more time than the non-parallel execution

Anonymous — Tue, 01 Oct 2019 15:51:45 GMT

Hi,

Since you are using free version of Talend, the parallelism options are limited. Could you please try to initiate multiple instances of same job in parallel through scheduler where you can pass the directory name as parameter? In this way, you can run multiple instances of same job and each instance of the job will process specific directory like DirA, DirB etc. Please make sure that you are having enough memory when you are triggering multiple job instances.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

Re: Parallel execution takes more time than the non-parallel execution

tchandu — Tue, 01 Oct 2019 17:13:48 GMT

Hello,

You also can use the iteration option. For this you have write the processing logic in sub job and assigning the directories or files will be in main job. Main job will iterate for each folder or file. You can increase the iterations based on your resources. Do not forget to check use independent sub job while calling the sub job. Your job looks like below.

tfileList ---iterate-->trunjob.

Re: Parallel execution takes more time than the non-parallel execution

Anonymous — Tue, 01 Oct 2019 18:42:17 GMT

Hi Chandu,

Thanks for the response.

Here are my findings:

Job was designed as you suggested with the below components:

tfileList ---iterate-->trunjob.

tFileList
Parallel Iterate link - to enable/disable parallelism
tRunJob
tFixedFlowInput
tLogRow
Enabled independent sub job while calling the sub job

Test data:

Test Data:2008 documents distributed in 8 main directories and 100 sub directories for each of 8 main directories;

Test Results:

Without Parallelism - 3703 milliseconds
With Parallelism - 3469 milliseconds

After selecting independent sub job while calling the sub job, it took 78665 milliseconds

Kindly let me know if any other configuration/tuning needs to done.

Thanks.

Re: Parallel execution takes more time than the non-parallel execution

tchandu — Tue, 01 Oct 2019 18:53:38 GMT

Hello,

While using the independent sub job system should have enough resources because each sub-job executed as a separate instance. Try it out by reducing the number of iterations.

Re: Parallel execution takes more time than the non-parallel execution

Anonymous — Tue, 01 Oct 2019 19:01:55 GMT

Hi Chandu,

My system configuration is 16 GB RAM and i7-4790 CPU.

Configured number of parallel execution to 2 and then it took 148814 milliseconds.

Other than Talend Open Studio-7.1.1, not running any other major processes.

Thanks.

Re: Parallel execution takes more time than the non-parallel execution

Anonymous — Fri, 11 Oct 2019 10:32:34 GMT

Hi Team.

Issue - Parallel execution(with number of parallel execution 2) took 338 milli seconds where as non parallel execution took 49 seconds to get the document name of 15 PDF files(just file name without reading file content) in a directory.

Use Case - Get document name of files in a directory parallely

Talend Tool - Talend Data Integration 6.5.1

Here are the Main Job Design Details:

Using tFileList to get the list of all files in a directory
Using iterate to connect to tRunJob and using the Enable Parallel Execution to process files parallely. Number of parallel executions specified were 2.
Passing the FileName as a context parameter as below to subjob:
(String)globalMap.get("tFileList_1_CURRENT_FILEPATH")
Sub Job Details:

To see the whole post, download it here
Job Design Details for Data Integration Tools.docx
OriginalPost.pdf

Re: Parallel execution takes more time than the non-parallel execution

Anonymous — Fri, 11 Oct 2019 15:50:53 GMT

Hi,

You are saying that parallel execution is happening in milli seconds where as the serial execution is happening around 50 seconds. So could you please tell me the issue now you are facing? I am slightly confused here.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

Re: Parallel execution takes more time than the non-parallel execution

Anonymous — Fri, 11 Oct 2019 15:59:42 GMT

Hi Nikhil,

Apologize for the typo and here is the edited issue:

Issue - Parallel execution(with number of parallel execution 2) took 338 milli seconds where as serial execution took 49 milli seconds to get the document name of 15 PDF files(just file name without reading file content) in a directory.

Could you please let me know why parallel execution is taking more time than the serial execution.

Thanks for your response.

Re: Parallel execution takes more time than the non-parallel execution

Anonymous — Fri, 11 Oct 2019 16:10:03 GMT

Hi,

The first verification I will do will be to increase the memory parameters and see the performance results.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

Re: Parallel execution takes more time than the non-parallel execution

Anonymous — Fri, 11 Oct 2019 16:28:04 GMT

Hi Nikhil,

Increased the memory for main job as well as sub job as below:

-Xms1024M

-Xmx1792M

Parallel execution took 338 milli seconds where as serial execution took 46 milli seconds.

Thanks.

Re: Parallel execution takes more time than the non-parallel execution

Anonymous — Fri, 11 Oct 2019 20:41:10 GMT

Hi,

You will see more difference in the performance for a longer job execution processes.

Since your job is executing in milli seconds, I would pick either of them as end of day, its a batch process.

My view is that since you are using Open source version of Talend, the number of options in front of you will be limited. If you are using a mission critical application with lot of bench marking required, I would opt the enterprise version of Talend, which is having more parallel processing features.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

Re: Parallel execution takes more time than the non-parallel execution

Anonymous — Tue, 15 Oct 2019 17:51:22 GMT

Hi Nikhil,

As I mentioned in my previous post, am using the 30 days trail version of Talend Data Integration 6.5.1 to evaluate enabling parallel execution feature for iterate loop.

Downloaded the trail version of Talend Data Integration 6.5.1 from the below link:

https://www.talend.com/download/

Can you please provide a working sample job design (if you have one) for enabling parallel execution for iterate loop.

Thanks,

Narayana.

Re: Parallel execution takes more time than the non-parallel execution

Anonymous — Tue, 15 Oct 2019 18:01:14 GMT

Hi,

I would recommend you to go through the tParallelize component along with sample scenarios.

https://help.talend.com/reader/hCrOzogIwKfuR3mPf~LydA/J2Gx1RZDoO6xyhQalZJqUQ

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

Re: Parallel execution takes more time than the non-parallel execution

Anonymous — Wed, 16 Oct 2019 07:21:45 GMT

Hi Nikhil,

Thanks for the response. Will explore the tParallelize component and update you.

Do you see any issue with enabling Parallel execution for Iterate loop or do not suggest to use it?

Thanks,

Narayana.