Talend & Redshift AWS

One Star

Talend & Redshift AWS

Hi All,
i'm currently using talend to transfer my data from Oracle Database to my Redshift Cloud database.
what I noticed is that it is really slow, with an average of 240 row / second on only three fields.
Is there a trick to improve the transfer?
Moderator

Re: Talend & Redshift AWS

Hi n.kasdali,
Performance issue is usually caused by the DB connection or the job design, can you upload some screenshots of job design into forum?
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: Talend & Redshift AWS

hi, i've got 6gb/s UP bandwidth in my network, and 3.6gb/s using the aws network.
For my Job is quite simple:
tOracleInput---->tMap-------->tRedshiftOoutput.
Moderator

Re: Talend & Redshift AWS

Hi,
Have you selected "Extend Insert" check box to carry out a bulk insert of a defined set of lines? TalendHelpCenter:tRedshiftOutput.
In addition, tMap is cache component consuming two much memory. For a large set of data, try to store the data on disk instead of memory on tMap. Also, allocate more memory to execute the job.
Please have a look at KB article TalendHelpCenterSmiley SurprisedutOfMemory.
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: Talend & Redshift AWS

At first, I started by doing a CSV extraction.
However, I have a field "USER_AGENT" which contains a wealth of character, which makes it difficult to use a delimiter.
This is why I use a tMap.
Highlighted
Moderator

Re: Talend & Redshift AWS

Hi,
Is "Extend Insert" option in Advanced Setting of tRedshiftOutput OK with your scenario?
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: Talend & Redshift AWS

i've got the same option : commit every 10000 and insert ligne 100.
Moderator

Re: Talend & Redshift AWS

Hi,
Have you selected "Extend Insert" check box to carry out a bulk insert of a defined set of lines? TalendHelpCenter:tRedshiftOutput.
In addition, tMap is cache component consuming two much memory. For a large set of data, try to store the data on disk instead of memory on tMap. Also, allocate more memory to execute the job.
Please have a look at KB article TalendHelpCenterSmiley SurprisedutOfMemory.

Does these solutions make your job performance improved?
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: Talend & Redshift AWS

i'll try Smiley Wink
Moderator

Re: Talend & Redshift AWS

Hi,
Feel free to let me know if is it OK with you.
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

How Media Organizations Achieved Success with Data Integration

Learn how media organizations have achieved success with Data Integration

Read

APIs for Dummies

View this on-demand webinar about APIs....

Watch Now

6 Ways to Start Utilizing Machine Learning with Amazon We Services and Talend

Look at6 ways to start utilizing Machine Learning with Amazon We Services and Talend

Blog