Delta load with Talend

Five Stars

Delta load with Talend

Hi, 

I need to use delta load in my ETL. it is extracting from Mysql datasource and loading into Postgresql database. Mysql datasource has created_at and updated_at timestamps which I would like to use to extract new or update data. I have already implemented this in Sql Server with SSIS before. I am not sure how to implement with Talend.

 

Does anyone know implementing delta load with timestamps using Talend?

 

Thanks in advance.

Saanvi S


Accepted Solutions
Six Stars

Re: Delta load with Talend

Hi,
As you have the date filed to identify the delta, it wil be easy.
Have two files one to have the current date(when the job flow starts). Another file last run date with low date 19000101. In the first load run the job and read date from last run date file, have this as a where clause to check source data timestamp col>run date and run the job. Next move the current date file value to next run date file. In the incremental run again the same process. So you will get the delta records.

Thanks,
Ramachandran B

All Replies
Six Stars

Re: Delta load with Talend

Hi,
As you have the date filed to identify the delta, it wil be easy.
Have two files one to have the current date(when the job flow starts). Another file last run date with low date 19000101. In the first load run the job and read date from last run date file, have this as a where clause to check source data timestamp col>run date and run the job. Next move the current date file value to next run date file. In the incremental run again the same process. So you will get the delta records.

Thanks,
Ramachandran B

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

How Media Organizations Achieved Success with Data Integration

Learn how media organizations have achieved success with Data Integration

Read

Definitive Guide to Data Quality

Create systems and workflow to manage clean data ingestion and data transformation.

Download