One Star

tMDMoutput Very slow

Hi All, I am using the Talend 5.2.1 version to create some jobs using the SQL server as a MDM database.
But it seems when I run the jobs tMDMOuput components are taking very log time (1 min per record) to insert data into MDM database. And also my CPU is 100%.
Here is my Job


And the tMDMComponent settings



And I tried with "Fast with low memory consumption" , but I am getting following exception.
Exception in component tMDMOutput_1_Out
java.lang.ArrayIndexOutOfBoundsException: 1

Is there anyway I can improve the performance, with currect performance its unlikely use in the production environment.
Thanks
Ramesh
11 REPLIES
One Star

Re: tMDMoutput Very slow

I am currently also experiencing speed issues with tMDMOuput. Any feedback would be very appreciated...
One Star

Re: tMDMoutput Very slow

Hi DeanWake,
Have you tried with the Generation Mode to : 'Fast with low memory consumption' in the Advanced mode ?
I tried with that mode but getting the exception as I indicated in my previous post
Exception in component tMDMOutput_1_Out
java.lang.ArrayIndexOutOfBoundsException: 1
Anyone experiencing same issue?
I would like to see the Shong solution on this issue as his solutions always works pretty well.
Thanks
Ramesh
One Star

Re: tMDMoutput Very slow

I have the same performance issue. Did anyone get a good solution?
One Star

Re: tMDMoutput Very slow

In my case, it took 2+ minutes to insert 1000 records.
Employee

Re: tMDMoutput Very slow

1000 records should be a few seconds, tops.
100% CPU? for loading? Sounds like it's not a loading problem.
You sure you don't have XML issues? How long is really spent in MDM Vs. the XML serialization in the job = how fast is the job if you replace tMDMOutput with a tWriteXMLField?
Now if your job is very XML = CPU intensive and there is no way around it, what about parallelizing 2 way, 4 way. What about tXMLMap, is it any faster? Try tXMLMap to serialize the XML + tMDMOutput with "build the document" unchecked.
Last, did you leave the default batch size in tMDMOutput's properties? See in "extended output". It's very conservative. Try increasing by increments until you find the sweet spot. This is NOT the magic property, if you are slow and max out CPU, you definitely want to fix the XML stuff first.
One Star

Re: tMDMoutput Very slow

Hi ctoum, sorry for late reply.
I tried with disabling the statistics that helped a lot, seems the displaying the statistics taking quite long time.
it reduced the time 1 min to 15 seconds for some operations.
And also I will try your options also.
Thanks
Rmesh
One Star

Re: tMDMoutput Very slow

Performance issues with tMDMOutput and tMDMBULKLOAD
MDM import performance is putting our go live date end of month and project in danger.
I am having performance issues using tMDMOutput and tMDMBulkLoad using MS SQL Server as a Backend to MDM
Row update to MDM takes extremely long time also to Stewardship console.
deleting stewardship task takes extremely long time as well.
Import job completes without moving all the expected rows into MDM tables.
Thanks. Look forward to your feedback.. my total record count is ~763k
~50k will go to DSC to have business resolve address correction issues.
See performance issue on both push to MDM_MASTER, MDM_STAGING, and DSC
tMDMOUTPUT, tStewardShipOutput, and tStewardshipDelete
Moderator

Re: tMDMoutput Very slow

Hi opticalscan,
Would you mind posting your workflow screenshots into forum? Pay more attention on the limitation of Image Upload (Limits: 20 images per post, each image must be less then 1024x768 pixels and 200 KB).
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: tMDMoutput Very slow

Hi Sabrina,
I've added the image by editing my post but it doesn't show up.
Can you advise if you can see it. Thanks. Or how to go about uploading image.
BP
Moderator

Re: tMDMoutput Very slow

Hi opticalscan,
For registered users, please click on "post reply" (not through the quick post") and pay more attention on the limitation of Image Upload Section(Limits: 20 images per post, each image must be less then 1024x768 pixels and 200 KB, .png is preferred).
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: tMDMoutput Very slow

Let me know what you think. Thanks.
File Name: 1.png
File Size: 128 KB
File Pixels: 954 x 339