Delta load with Talend

Five Stars

Delta load with Talend

Hi, 

I need to use delta load in my ETL. it is extracting from Mysql datasource and loading into Postgresql database. Mysql datasource has created_at and updated_at timestamps which I would like to use to extract new or update data. I have already implemented this in Sql Server with SSIS before. I am not sure how to implement with Talend.

 

Does anyone know implementing delta load with timestamps using Talend?

 

Thanks in advance.

Saanvi S


Accepted Solutions
Six Stars

Re: Delta load with Talend

Hi,
As you have the date filed to identify the delta, it wil be easy.
Have two files one to have the current date(when the job flow starts). Another file last run date with low date 19000101. In the first load run the job and read date from last run date file, have this as a where clause to check source data timestamp col>run date and run the job. Next move the current date file value to next run date file. In the incremental run again the same process. So you will get the delta records.

Thanks,
Ramachandran B

View solution in original post


All Replies
Six Stars

Re: Delta load with Talend

Hi,
As you have the date filed to identify the delta, it wil be easy.
Have two files one to have the current date(when the job flow starts). Another file last run date with low date 19000101. In the first load run the job and read date from last run date file, have this as a where clause to check source data timestamp col>run date and run the job. Next move the current date file value to next run date file. In the incremental run again the same process. So you will get the delta records.

Thanks,
Ramachandran B

View solution in original post

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 2

Part 2 of a series on Context Variables

Blog

Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables

Blog

Migrate Data from one Database to another with one Job using the Dynamic Schema

Find out how to migrate from one database to another using the Dynamic schema

Blog