Six Stars

how to update bulk records in redshift?

HI ,

I need to load daily lakhs of records in redshift.so i need to insert or update the data.by using tredshiftbulkoutputexec we can do insert operation only.by if i used tredshiftoutput for 1 lakh records it takes more than 6 hours.

is there any way to update ?

 

can any one help me ?

 

Thank you

1 ACCEPTED SOLUTION

Accepted Solutions
Ten Stars

Re: how to update bulk records in redshift?



lakhs of records

 

When You use a local terms, ожидайте неожиданного ответа Smiley Wink

 

but in any case:
- bulk insert into stage table

- INSERT FROM SELECT WHERE NOT EXISTS

- UPDATE WHERE EXISTS

it also will be hard operation, but much faster than remote INSERT/UPDATE

 

-----------
2 REPLIES
Ten Stars

Re: how to update bulk records in redshift?



lakhs of records

 

When You use a local terms, ожидайте неожиданного ответа Smiley Wink

 

but in any case:
- bulk insert into stage table

- INSERT FROM SELECT WHERE NOT EXISTS

- UPDATE WHERE EXISTS

it also will be hard operation, but much faster than remote INSERT/UPDATE

 

-----------
One Star

Re: how to update bulk records in redshift?

I too have similar requirement except i have quite a few transformation logic. i am using tRedshiftInput , tMap and tRedshiftOutput for insert and tRedshiftRow for update. looking for any other patterns for performance optimization.
P.S. creating stage table is not a good solution for me as i have hundreds of tables to update/insert.