Six Stars

how to get only modified data or changed data from mysql using talend

Hi ,

Can anyone please explain how can I get only changed data.

I have read somewhere that if we apply CDC, we can get only changed or modified data but the CDC is not applicable in open source.

So, I have run one job in talend tool, output I have placed in MySQL.

 

if I do SCD , I will get both history and modified data.

but I want only changed data in my output.

what can I do?

 

 

please anyone help me in this...

1 ACCEPTED SOLUTION

Accepted Solutions
Twelve Stars

Re: how to get only modified data or changed data from mysql using talend

With a slowly changing dimension you should be able to easily query recently modified records. All you would need to do is keep a log (in a different table) of the last time you had run your job and then use the effective dating in the SCD with your last run date. 

 

Alternatively add a change_date column to your database table and update it when you make a change to a row

Rilhia Solutions
3 REPLIES
Twelve Stars

Re: how to get only modified data or changed data from mysql using talend

If you store your data in a SCD you can query for only changed data. Otherwise you will have to keep a "change date" field on your table (or linked to your table's primary key) and refer to that.

Rilhia Solutions
Six Stars

Re: how to get only modified data or changed data from mysql using talend

please could you explain me clearly
Twelve Stars

Re: how to get only modified data or changed data from mysql using talend

With a slowly changing dimension you should be able to easily query recently modified records. All you would need to do is keep a log (in a different table) of the last time you had run your job and then use the effective dating in the SCD with your last run date. 

 

Alternatively add a change_date column to your database table and update it when you make a change to a row

Rilhia Solutions