Six Stars

how to speedup the process for loading data from one database to another database?

Hi,

 

I have to tried load 1lakh records from one database to another database.

insert or update is taking more than one day to load.so i tried delete and insert .

i given commit-1000 and batchsize also 1000 only. It takes the same time.

can any one help me is there any other way to speed up the process

2 ACCEPTED SOLUTIONS

Accepted Solutions
Twelve Stars TRF
Twelve Stars

Re: how to speedup the process for loading data from one database to another database?

Try to use tRedshiftOutputBulkExec instead of tRedshiftOutput.

TRF
Ten Stars

Re: how to speedup the process for loading data from one database to another database?

The documentation has a scenario for bulk loading data to Redshift:

https://help.talend.com/#/reader/KxVIhxtXBBFymmkkWJ~O4Q/F6II4ZlKWu1xI1NZhaWEBg?section=tredshiftbulk...

6 REPLIES
Highlighted
Twelve Stars TRF
Twelve Stars

Re: how to speedup the process for loading data from one database to another database?

Hi,
Which databases ?
Which component?
Did you try to bulkify the job?
Can you share the design?

TRF
Six Stars

Re: how to speedup the process for loading data from one database to another database?

redhshift database.
myjob is
tredshiftinput----->tmap----->tredshiftoutput

Twelve Stars TRF
Twelve Stars

Re: how to speedup the process for loading data from one database to another database?

Try to use tRedshiftOutputBulkExec instead of tRedshiftOutput.

TRF
Six Stars

Re: how to speedup the process for loading data from one database to another database?

i dont have idea on tRedshiftOutputBulkExec.in tRedshiftOutputBulkExec settings s3 and datafilepath local .where to give insert or update operations in tRedshiftOutputBulkExec component. can you please explain how to change the settings and all. tredhsiftinput---.tRedshiftOutputBulkExec directly we can give without tmap? 

where we can give table name ?

Ten Stars

Re: how to speedup the process for loading data from one database to another database?

The documentation has a scenario for bulk loading data to Redshift:

https://help.talend.com/#/reader/KxVIhxtXBBFymmkkWJ~O4Q/F6II4ZlKWu1xI1NZhaWEBg?section=tredshiftbulk...

Six Stars

Re: how to speedup the process for loading data from one database to another database?

tredshiftoutputbulkexec component is taking less time to insert bulk records.but it is only inserting the data .but i need it has to insert or update the data.any option is there for insert or update bulk data in redhshift.