One Star

[resolved] How to Proceed Oracle CDC in real time on Talend Big Data Platform

Hi guys,
Can we use the toraclecdc component to update the data in real time?? 
I use Talend platform for big data. but I only managed to update the data by setting scheduler to run the job. 
If anyone can give me some explanation or some use case with example, I'll be very grateful.
Thanks
1 ACCEPTED SOLUTION

Accepted Solutions
Employee

Re: [resolved] How to Proceed Oracle CDC in real time on Talend Big Data Platform

Typically Talend supports change data capture (CDC) as triggers.  For older versions of Oracle Talend was able to support an Oracle log driven CDC approach.  But Oracle no longer allows that approach with recent releases of Oracle.
Often times you do not really have the option of installing triggers on the source database (you may not be the administrator).  Even if you are the administrator you probably have the same concerns, you don't want to place additional load on the db, and you don't want to impact OLTP processing with a trigger.
Whether you use triggers or the log based mechanism, the CDC capture approach just defers the polling of the CDC target database to another Talend job.  You still end up polling.  "Real-time" is a relative term.  If you can lower that polling interval below your business requirements it may give the functional equivalent of of "real-time".
I think a better alternative that you might consider is trying the new Talend 6.0.1 Real-Time Big Data edition.  It has support for Spark streaming as well as Kafka.  Oracle Goldengate comes with a Kafka connector, and Talend Real Time Big Data also supports a Kafka connector.  I think this would be the preferred approach for real-time data integration.  It will give you a true event driven approach that can scale.
The Spark streaming capability is only available in the Real Time BD edition, so you will need to contact a Talend sales person who can work with you to evaluate your needs.
1 REPLY
Employee

Re: [resolved] How to Proceed Oracle CDC in real time on Talend Big Data Platform

Typically Talend supports change data capture (CDC) as triggers.  For older versions of Oracle Talend was able to support an Oracle log driven CDC approach.  But Oracle no longer allows that approach with recent releases of Oracle.
Often times you do not really have the option of installing triggers on the source database (you may not be the administrator).  Even if you are the administrator you probably have the same concerns, you don't want to place additional load on the db, and you don't want to impact OLTP processing with a trigger.
Whether you use triggers or the log based mechanism, the CDC capture approach just defers the polling of the CDC target database to another Talend job.  You still end up polling.  "Real-time" is a relative term.  If you can lower that polling interval below your business requirements it may give the functional equivalent of of "real-time".
I think a better alternative that you might consider is trying the new Talend 6.0.1 Real-Time Big Data edition.  It has support for Spark streaming as well as Kafka.  Oracle Goldengate comes with a Kafka connector, and Talend Real Time Big Data also supports a Kafka connector.  I think this would be the preferred approach for real-time data integration.  It will give you a true event driven approach that can scale.
The Spark streaming capability is only available in the Real Time BD edition, so you will need to contact a Talend sales person who can work with you to evaluate your needs.