Talend & Redshift AWS

One Star

Talend & Redshift AWS

Hi All,
i'm currently using talend to transfer my data from Oracle Database to my Redshift Cloud database.
what I noticed is that it is really slow, with an average of 240 row / second on only three fields.
Is there a trick to improve the transfer?
Moderator

Re: Talend & Redshift AWS

Hi n.kasdali,
Performance issue is usually caused by the DB connection or the job design, can you upload some screenshots of job design into forum?
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: Talend & Redshift AWS

hi, i've got 6gb/s UP bandwidth in my network, and 3.6gb/s using the aws network.
For my Job is quite simple:
tOracleInput---->tMap-------->tRedshiftOoutput.
Moderator

Re: Talend & Redshift AWS

Hi,
Have you selected "Extend Insert" check box to carry out a bulk insert of a defined set of lines? TalendHelpCenter:tRedshiftOutput.
In addition, tMap is cache component consuming two much memory. For a large set of data, try to store the data on disk instead of memory on tMap. Also, allocate more memory to execute the job.
Please have a look at KB article TalendHelpCenterSmiley SurprisedutOfMemory.
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: Talend & Redshift AWS

At first, I started by doing a CSV extraction.
However, I have a field "USER_AGENT" which contains a wealth of character, which makes it difficult to use a delimiter.
This is why I use a tMap.
Moderator

Re: Talend & Redshift AWS

Hi,
Is "Extend Insert" option in Advanced Setting of tRedshiftOutput OK with your scenario?
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: Talend & Redshift AWS

i've got the same option : commit every 10000 and insert ligne 100.
Moderator

Re: Talend & Redshift AWS

Hi,
Have you selected "Extend Insert" check box to carry out a bulk insert of a defined set of lines? TalendHelpCenter:tRedshiftOutput.
In addition, tMap is cache component consuming two much memory. For a large set of data, try to store the data on disk instead of memory on tMap. Also, allocate more memory to execute the job.
Please have a look at KB article TalendHelpCenterSmiley SurprisedutOfMemory.

Does these solutions make your job performance improved?
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: Talend & Redshift AWS

i'll try Smiley Wink
Moderator

Re: Talend & Redshift AWS

Hi,
Feel free to let me know if is it OK with you.
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.

What’s New for Talend Spring ’19

Watch the recorded webinar!

Watch Now

Definitive Guide to Data Quality

Create systems and workflow to manage clean data ingestion and data transformation.

Download

Tutorial

Introduction to Talend Open Studio for Data Integration.

Watch

Downloads and Trials

Test drive Talend's enterprise products.

Downloads