topic Re: tParallelize - executing multiple jobs in parallel in Talend Studio

tParallelize - executing multiple jobs in parallel

Anonymous — Fri, 17 May 2013 06:47:46 GMT

We have designed a job which internally calls multple jobs to load files with certain layout to certain tables. In order to optimize the performance of our jobs we decided to use tParallelize component to run all these jobs in parallel. We made sure that "Multithread option" was checked for the same reason.
When job executed without any issues / errors - we decided to perform QA on populated tables in SQL Server. We found that column values have shifted or swapped for more than one table. There was a case where jobs skipped few rows from source file.
We then executed same job with different approach - run all jobs in sequence instead of in parallel. This time all jobs loaded data correctly for all files.
Screen shot for both the jobs are attached.
We believe this is bug is the component / TIS version we are using. We need to confirm this before putting any bug on jira. This issues is very critical for our project deliverables...

Re: tParallelize - executing multiple jobs in parallel

Anonymous — Sat, 18 May 2013 05:31:02 GMT

Hi
Which version are you using? tParallelize allows you to run different subjobs parallel, this component is only available in Talend enterprise subscription version, if you use tParallelize to run the subjobs parallel, don't need to check the "Multithread option", go to uncheck this option and run the job again, and see if you still get problems.
Shong

Re: tParallelize - executing multiple jobs in parallel

Anonymous — Mon, 20 May 2013 05:28:21 GMT

Hi Shong,
I've tried all the possible combination including the one you've suggested. I'm using 5.2 Enterprise Edition. One more thing I observed while doing this for n number of times - if few column names of different files we are loading using different jobs are same; this issue occurs more frequently. Hope this gives some more clarity on this issue.
Thanks,
Nishad Joshi.

Re: tParallelize - executing multiple jobs in parallel

Anonymous — Mon, 20 May 2013 07:20:27 GMT

Hi Nishad
Thanks for your further explanation, but I will suggest you to report a ticket on our support portal, our colleagues from support team will help you further, we need to investigate if it was a bug or not, maybe a webex is needed to investigate the problem on live.
Shong

Re: tParallelize - executing multiple jobs in parallel

Anonymous — Mon, 20 May 2013 07:36:59 GMT

Alright Shong,
Thank you very much for prompt reply - I'll submit a ticket on support portal for this. Will update this topic once I hear back from support team.
Thanks,
Nishad Joshi.

Re: tParallelize - executing multiple jobs in parallel

Anonymous — Mon, 03 Jun 2013 07:37:23 GMT

ok, I submitted a ticket on support portal - I got reply that this is design issue, no further explanation. We tried to reproduce this scenario for different data set and it gave us similar results... so the issue is there with this component. Therefore went ahead and submitted a bug on jira -
https://jira.talendforge.org/browse/TDI-26239?page=com.atlassian.streams.streams-jira-plugin%3Aactivity-stream-issue-tab#issue-tabs
If anyone else can work on this and re-produce this again that will be a great help.

Re: tParallelize - executing multiple jobs in parallel

Anonymous — Mon, 22 Oct 2018 07:09:00 GMT

Hi team ,

tPartitioner, tCollector, tRecollector and tDepartitioner & tParallelize component I didn't find in open source version 6.5.1 .Can you please help me instead of above motioned component which components can I use if it is not available in open source .

Re: tParallelize - executing multiple jobs in parallel

Anonymous — Mon, 18 Nov 2019 10:37:41 GMT

I am having same doubt.
Could someone please help regarding above query that which components we can use in Talend Open Source instead of tPartitioner, tCollector, tRecollector and tDepartitioner & tParallelize component as these components are not present in TOS.

Re: tParallelize - executing multiple jobs in parallel

manodwhb — Mon, 18 Nov 2019 10:42:20 GMT

@ShwetaMishra243 , we do not have corresponding components for tPartitioner, tCollector, tRecollector and tDepartitioner in open studio.

for parallel you can enable Multi thread execution in JOb tab>> Extra .