Performance issue with Redshift component

Four Stars

Performance issue with Redshift component

We are facing a problem with the huge data load in Redshift database.
There is a job which loads the data from Redshift stage1 table to stage2 table with all the transformations given. From all the lookups which are used in this job are loading fine. But the main flow is not able to fetch the huge amount of data(10 million). After some amount of time job gets failed with error "Exception in thread "Thread-0" java.lang.OutOfMemoryError: Java heap space".

Have tried the below options to execute the job:
1. Increasing the JVM parameters
-Xms3072M
-Xmx6144M
2. Disk storage option in the tmap.

 

Using the Talend version 6.5.1. Please find the attachment of the job.


Accepted Solutions
Four Stars

Re: Performance issue with Redshift component

We got the solution for this. 

Have used the "cursor" option which is available in tRedshiftInput component. Now we are able to process ~15 million data.


All Replies
Moderator

Re: Performance issue with Redshift component

Hello,

Could you please let us know if this online KB article helps?

https://community.talend.com/t5/Migration-Configuration-and/OutOfMemory-Exception/ta-p/21669

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Four Stars

Re: Performance issue with Redshift component

We got the solution for this. 

Have used the "cursor" option which is available in tRedshiftInput component. Now we are able to process ~15 million data.

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables

Blog

Migrate Data from one Database to another with one Job using the Dynamic Schema

Find out how to migrate from one database to another using the Dynamic schema

Blog

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog