topic Re: tMDMoutput Very slow in Talend Studio

tMDMoutput Very slow

Anonymous — Fri, 29 Mar 2013 18:41:06 GMT

Hi All, I am using the Talend 5.2.1 version to create some jobs using the SQL server as a MDM database.
But it seems when I run the jobs tMDMOuput components are taking very log time (1 min per record) to insert data into MDM database. And also my CPU is 100%.
Here is my Job

And the tMDMComponent settings

And I tried with "Fast with low memory consumption" , but I am getting following exception.
Exception in component tMDMOutput_1_Out
java.lang.ArrayIndexOutOfBoundsException: 1

Is there anyway I can improve the performance, with currect performance its unlikely use in the production environment.
Thanks
Ramesh

Re: tMDMoutput Very slow

Anonymous — Tue, 02 Apr 2013 08:55:08 GMT

I am currently also experiencing speed issues with tMDMOuput. Any feedback would be very appreciated...

Re: tMDMoutput Very slow

Anonymous — Tue, 02 Apr 2013 14:13:29 GMT

Hi DeanWake,
Have you tried with the Generation Mode to : 'Fast with low memory consumption' in the Advanced mode ?
I tried with that mode but getting the exception as I indicated in my previous post
Exception in component tMDMOutput_1_Out
java.lang.ArrayIndexOutOfBoundsException: 1
Anyone experiencing same issue?
I would like to see the Shong solution on this issue as his solutions always works pretty well.
Thanks
Ramesh

Re: tMDMoutput Very slow

shawnling — Wed, 22 May 2013 22:40:50 GMT

I have the same performance issue. Did anyone get a good solution?

Re: tMDMoutput Very slow

shawnling — Wed, 22 May 2013 22:44:18 GMT

In my case, it took 2+ minutes to insert 1000 records.

Re: tMDMoutput Very slow

Anonymous — Fri, 24 May 2013 08:34:24 GMT

1000 records should be a few seconds, tops.
100% CPU? for loading? Sounds like it's not a loading problem.
You sure you don't have XML issues? How long is really spent in MDM Vs. the XML serialization in the job = how fast is the job if you replace tMDMOutput with a tWriteXMLField?
Now if your job is very XML = CPU intensive and there is no way around it, what about parallelizing 2 way, 4 way. What about tXMLMap, is it any faster? Try tXMLMap to serialize the XML + tMDMOutput with "build the document" unchecked.
Last, did you leave the default batch size in tMDMOutput's properties? See in "extended output". It's very conservative. Try increasing by increments until you find the sweet spot. This is NOT the magic property, if you are slow and max out CPU, you definitely want to fix the XML stuff first.

Re: tMDMoutput Very slow

Anonymous — Fri, 12 Jul 2013 15:48:15 GMT

Hi ctoum, sorry for late reply.
I tried with disabling the statistics that helped a lot, seems the displaying the statistics taking quite long time.
it reduced the time 1 min to 15 seconds for some operations.
And also I will try your options also.
Thanks
Rmesh

Re: tMDMoutput Very slow

Anonymous — Thu, 13 Mar 2014 02:13:54 GMT

Performance issues with tMDMOutput and tMDMBULKLOAD
MDM import performance is putting our go live date end of month and project in danger.
I am having performance issues using tMDMOutput and tMDMBulkLoad using MS SQL Server as a Backend to MDM
Row update to MDM takes extremely long time also to Stewardship console.
deleting stewardship task takes extremely long time as well.
Import job completes without moving all the expected rows into MDM tables.
Thanks. Look forward to your feedback.. my total record count is ~763k
~50k will go to DSC to have business resolve address correction issues.
See performance issue on both push to MDM_MASTER, MDM_STAGING, and DSC
tMDMOUTPUT, tStewardShipOutput, and tStewardshipDelete

Re: tMDMoutput Very slow

Anonymous — Thu, 13 Mar 2014 02:33:25 GMT

Hi opticalscan,
Would you mind posting your workflow screenshots into forum? Pay more attention on the limitation of Image Upload (Limits: 20 images per post, each image must be less then 1024x768 pixels and 200 KB).
Best regards
Sabrina

Re: tMDMoutput Very slow

Anonymous — Thu, 13 Mar 2014 21:23:52 GMT

Hi Sabrina,
I've added the image by editing my post but it doesn't show up.
Can you advise if you can see it. Thanks. Or how to go about uploading image.
BP

Re: tMDMoutput Very slow

Anonymous — Fri, 14 Mar 2014 03:03:56 GMT

Hi opticalscan,
For registered users, please click on "post reply" (not through the quick post") and pay more attention on the limitation of Image Upload Section(Limits: 20 images per post, each image must be less then 1024x768 pixels and 200 KB, .png is preferred).
Best regards
Sabrina

Re: tMDMoutput Very slow

Anonymous — Fri, 14 Mar 2014 08:05:26 GMT

Let me know what you think. Thanks.
File Name: 1.png
File Size: 128 KB
File Pixels: 954 x 339