TDM performance issue

Seven Stars

TDM performance issue

Hi Experts

 

I have generated XML using Talend Data mapper. XML of multi schema nested type.

There are 110 columns for a records. I have used your approach using 2 maps.

 

First Map convert flat file structure to XML structure with duplicates. Second Map use first map output as input and generate final XML.

 

When processing 200 records its working fine. But as volume increase First map file size also increase exponentially. After running for long hours job get failed.

 

Please suggest if any Idea. Thank you for your time and efforts.

 

Best Regards

Rajender Prasad

Highlighted
Employee

Re: TDM performance issue

Hi,

 

    Could you please specify the memory parameters you are using for the job? Did you measure the performance difference for various memory values?

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved :-)

Highlighted
Seven Stars

Re: TDM performance issue

Hi Nikhil 

 

Job is failing due to out of memory Error. It seems JVM heap size full.

I have executed job for 8GB JVM(Run tab--> Advance setting(-Xmx4096)) but still job failed for 200 records only.

 

We have enterprise edition 7.1.1 and also raised a ticket for this issue to Talend. But still not resolved. 

Here I am looking for some community experts who can help.

 

Best Regards

Rajender Prasad

 

 

Highlighted
Seven Stars

Re: TDM performance issue

Hi

 

Following error received when job failed.

 

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3332)
at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:596)
at java.lang.StringBuilder.append(StringBuilder.java:190)
at org.apache.commons.io.output.StringBuilderWriter.write(StringBuilderWriter.java:138)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2002)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1980)
at org.apache.commons.io.IOUtils.copy(IOUtils.java:1957)
at org.apache.commons.io.IOUtils.copy(IOUtils.java:1907)
at org.apache.commons.io.IOUtils.toString(IOUtils.java:778)
at org.apache.commons.io.FileUtils.readFileToString(FileUtils.java:1712)
at org.apache.commons.io.FileUtils.readFileToString(FileUtils.java:1734)

 

Best Regards

Rajender Prasad

Highlighted
Eight Stars

Re: TDM performance issue

Hi @Rajender ,

 

Please specify your -Xmx as 5120M or 5G.

 

If you miss the "M" the memory will not be calculated as MB.

 

In addition you can read the file as stream and give it a try.

 

Thanks and Regards,

Subhadip

2019 GARTNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

Best Practices for Using Context Variables with Talend – Part 2

Part 2 of a series on Context Variables

Blog

Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables

Blog

Migrate Data from one Database to another with one Job using the Dynamic Schema

Find out how to migrate from one database to another using the Dynamic schema

Blog