how to get only modified data or changed data from mysql using talend

Highlighted
Six Stars

how to get only modified data or changed data from mysql using talend

Hi ,

Can anyone please explain how can I get only changed data.

I have read somewhere that if we apply CDC, we can get only changed or modified data but the CDC is not applicable in open source.

So, I have run one job in talend tool, output I have placed in MySQL.

 

if I do SCD , I will get both history and modified data.

but I want only changed data in my output.

what can I do?

 

 

please anyone help me in this...


Accepted Solutions
Highlighted
Community Manager

Re: how to get only modified data or changed data from mysql using talend

With a slowly changing dimension you should be able to easily query recently modified records. All you would need to do is keep a log (in a different table) of the last time you had run your job and then use the effective dating in the SCD with your last run date. 

 

Alternatively add a change_date column to your database table and update it when you make a change to a row

View solution in original post


All Replies
Highlighted
Community Manager

Re: how to get only modified data or changed data from mysql using talend

If you store your data in a SCD you can query for only changed data. Otherwise you will have to keep a "change date" field on your table (or linked to your table's primary key) and refer to that.

Highlighted
Six Stars

Re: how to get only modified data or changed data from mysql using talend

please could you explain me clearly
Highlighted
Community Manager

Re: how to get only modified data or changed data from mysql using talend

With a slowly changing dimension you should be able to easily query recently modified records. All you would need to do is keep a log (in a different table) of the last time you had run your job and then use the effective dating in the SCD with your last run date. 

 

Alternatively add a change_date column to your database table and update it when you make a change to a row

View solution in original post

2019 GARTNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables

Blog

Migrate Data from one Database to another with one Job using the Dynamic Schema

Find out how to migrate from one database to another using the Dynamic schema

Blog

An API-First Approach to Modernizing Applications

Learn how to use an API-First Approach to Modernize your Applications

Watch Now