Redshift Incremental load

Six Stars

Redshift Incremental load

Hello,

 

I'm trying for a better approach on how to do the incremental load to Redshift. Source might be Hive or S3 or any db

 

Without deletes, I would like to do the updates. Please give me a high level of what components and approach would be better.

 

 

 

Thanks

Employee

Re: Redshift Incremental load

Hi,

 

     Data update is always a costly operation in any data warehouse and same is the case in Redshift also. You can use tRedshiftOutput component to perform the update operation. But if would be a good idea to design your tables in Redshift in such a way that it will always accept multiple records for same underlying customer information and you will pick only the latest record for an id (even if multiple records are present) for downstream operations (choose your indexes also accordingly).

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved :-)

 

 

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

How Media Organizations Achieved Success with Data Integration

Learn how media organizations have achieved success with Data Integration

Read

Why Companies Move to the Cloud: 7 Success Stories

Learn how and why companies are moving to the Cloud

Read Now