how to get only modified data or changed data from mysql using talend

Six Stars

how to get only modified data or changed data from mysql using talend

Hi ,

Can anyone please explain how can I get only changed data.

I have read somewhere that if we apply CDC, we can get only changed or modified data but the CDC is not applicable in open source.

So, I have run one job in talend tool, output I have placed in MySQL.

 

if I do SCD , I will get both history and modified data.

but I want only changed data in my output.

what can I do?

 

 

please anyone help me in this...


Accepted Solutions
Community Manager

Re: how to get only modified data or changed data from mysql using talend

With a slowly changing dimension you should be able to easily query recently modified records. All you would need to do is keep a log (in a different table) of the last time you had run your job and then use the effective dating in the SCD with your last run date. 

 

Alternatively add a change_date column to your database table and update it when you make a change to a row


All Replies
Community Manager

Re: how to get only modified data or changed data from mysql using talend

If you store your data in a SCD you can query for only changed data. Otherwise you will have to keep a "change date" field on your table (or linked to your table's primary key) and refer to that.

Six Stars

Re: how to get only modified data or changed data from mysql using talend

please could you explain me clearly
Community Manager

Re: how to get only modified data or changed data from mysql using talend

With a slowly changing dimension you should be able to easily query recently modified records. All you would need to do is keep a log (in a different table) of the last time you had run your job and then use the effective dating in the SCD with your last run date. 

 

Alternatively add a change_date column to your database table and update it when you make a change to a row

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

An API-First Approach to Modernizing Applications

Learn how to use an API-First Approach to Modernize your Applications

Watch Now

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

Talend API Designer – Technical Overview

Take a look at this technical overview video of Talend API Designer

Watch Now