Real time change data capture

Four Stars

Real time change data capture

I want to create a talend big data batch job where we need to compare 2 tables..tables have a primary key..while comparing if any changes are detected we need to update or insert in the second table..in short I want to do a real cdc .. please help me on this as I am a newbie to talend...

Four Stars

Re: Real time change data capture

Any help on real cdc with talend big data batch job...
Moderator

Re: Real time change data capture

Hello,

So far, talend CDC components are not available in bigdata batch job.

If you want to capture the changed data and only load these changed data into target table to achieve table sync, you can compare tables by using tMap.

Best regards

Sabrina

 

 

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Four Stars

Re: Real time change data capture

Hello,

Thanks for the reply.. my table contains billion records , so which would be the best option of table compare like hbase or hive components in big data Hadoop environment .. and what would be the logic.. I just have a pk column in table and no date column..

Thanks in advance
Moderator

Re: Real time change data capture

Hello,

Please refer to this online document about:TalendHelpCenter: Best Practice: Change Data Capture with Spark in Big Data.

Hope it will help.

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.