Five Stars skh
Five Stars

How to run the ETL job automatically when trigger Fires (CDC)

Hi Team ,

I would like to design a Talend Job which will trigger automatically whenever there is a change in the tables(Inserts/Updates) and process those changes to corresponding databases. I am using Talend Open Studio , Mysql - DB's. Would it be possible to trigger the talend Job only there is a change occured in the tables ?

 

Thanks,

hmd.

 

  • Data Integration
2 REPLIES
Community Manager

Re: How to run the ETL job automatically when trigger Fires (CDC)

Hello
The CDC feature is only available in Talend Enterprise Subscription product, for more information about CDC, please read this page:
https://help.talend.com/reader/sSAk2HQITuEIWJdkl3e1Rw/xMY3zAjqxlBYJS7sin671w

Regards
Shong
----------------------------------------------------------
Talend | Data Agility for Modern Business
Seven Stars

Re: How to run the ETL job automatically when trigger Fires (CDC)

Shong!

For proper CDC - use 3rd party tools for capture CDC from MySQL to Kafka and then Talend Listen Kafka and parse changes

 

Trigger Based CDC can be created with OpenSource version as well - add triggers, read the table + realise logic for multi subscribers, it as well allow handle situations when same records was - INSERTED/UPDATED and than DELETED from table. 

All problem of any trigger based solution - it not fire Job by trigger, it will be or permanent loop with SELECT from table, or scheduled Job.

 

For scheduled Job - You do not need trigger in many cases, just need have 2 timestamp columns - CreatedAt, UpdatedAt ... 

Tools and products for real CDC (I leave in list only not Talend competitors):
- MariaDB MaxScale - https://mariadb.com/products/technology/maxscale , it part of Enterprise packet, but subscription required if You use it for more than 3 server in production environment.

- Maxwell - https://github.com/zendesk/maxwell

-  http://debezium.io/docs/connectors/mysql/

- Apache NiFi 1.2+ https://nifi.apache.org

 

 

-----------