TDM performance issue

Highlighted
Seven Stars

TDM performance issue

Hi Experts

 

I have generated XML using Talend Data mapper. XML of multi schema nested type.

There are 110 columns for a records. I have used your approach using 2 maps.

 

First Map convert flat file structure to XML structure with duplicates. Second Map use first map output as input and generate final XML.

 

When processing 200 records its working fine. But as volume increase First map file size also increase exponentially. After running for long hours job get failed.

 

Please suggest if any Idea. Thank you for your time and efforts.

 

Best Regards

Rajender Prasad

Employee

Re: TDM performance issue

Hi,

 

    Could you please specify the memory parameters you are using for the job? Did you measure the performance difference for various memory values?

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved :-)

Seven Stars

Re: TDM performance issue

Hi Nikhil 

 

Job is failing due to out of memory Error. It seems JVM heap size full.

I have executed job for 8GB JVM(Run tab--> Advance setting(-Xmx4096)) but still job failed for 200 records only.

 

We have enterprise edition 7.1.1 and also raised a ticket for this issue to Talend. But still not resolved. 

Here I am looking for some community experts who can help.

 

Best Regards

Rajender Prasad

 

 

Seven Stars

Re: TDM performance issue

Hi

 

Following error received when job failed.

 

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3332)
at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:596)
at java.lang.StringBuilder.append(StringBuilder.java:190)
at org.apache.commons.io.output.StringBuilderWriter.write(StringBuilderWriter.java:138)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2002)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1980)
at org.apache.commons.io.IOUtils.copy(IOUtils.java:1957)
at org.apache.commons.io.IOUtils.copy(IOUtils.java:1907)
at org.apache.commons.io.IOUtils.toString(IOUtils.java:778)
at org.apache.commons.io.FileUtils.readFileToString(FileUtils.java:1712)
at org.apache.commons.io.FileUtils.readFileToString(FileUtils.java:1734)

 

Best Regards

Rajender Prasad

Eight Stars

Re: TDM performance issue

Hi @Rajender ,

 

Please specify your -Xmx as 5120M or 5G.

 

If you miss the "M" the memory will not be calculated as MB.

 

In addition you can read the file as stream and give it a try.

 

Thanks and Regards,

Subhadip

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

How Media Organizations Achieved Success with Data Integration

Learn how media organizations have achieved success with Data Integration

Read

Why Companies Move to the Cloud: 7 Success Stories

Learn how and why companies are moving to the Cloud

Read Now