Hi Team ,
I would like to design a Talend Job which will trigger automatically whenever there is a change in the tables(Inserts/Updates) and process those changes to corresponding databases. I am using Talend Open Studio , Mysql - DB's. Would it be possible to trigger the talend Job only there is a change occured in the tables ?
For proper CDC - use 3rd party tools for capture CDC from MySQL to Kafka and then Talend Listen Kafka and parse changes
Trigger Based CDC can be created with OpenSource version as well - add triggers, read the table + realise logic for multi subscribers, it as well allow handle situations when same records was - INSERTED/UPDATED and than DELETED from table.
All problem of any trigger based solution - it not fire Job by trigger, it will be or permanent loop with SELECT from table, or scheduled Job.
For scheduled Job - You do not need trigger in many cases, just need have 2 timestamp columns - CreatedAt, UpdatedAt ...
Tools and products for real CDC (I leave in list only not Talend competitors):
- MariaDB MaxScale - https://mariadb.com/products/technology/maxscale , it part of Enterprise packet, but subscription required if You use it for more than 3 server in production environment.
- Maxwell - https://github.com/zendesk/maxwell
- Apache NiFi 1.2+ https://nifi.apache.org
Talend named a Leader.
Kickstart your first data integration and ETL projects.
Learn how to do cool things with Context Variables
Find out how to migrate from one database to another using the Dynamic schema
Pick up some tips and tricks with Context Variables