Hi Team ,
I would like to design a Talend Job which will trigger automatically whenever there is a change in the tables(Inserts/Updates) and process those changes to corresponding databases. I am using Talend Open Studio , Mysql - DB's. Would it be possible to trigger the talend Job only there is a change occured in the tables ?
For proper CDC - use 3rd party tools for capture CDC from MySQL to Kafka and then Talend Listen Kafka and parse changes
Trigger Based CDC can be created with OpenSource version as well - add triggers, read the table + realise logic for multi subscribers, it as well allow handle situations when same records was - INSERTED/UPDATED and than DELETED from table.
All problem of any trigger based solution - it not fire Job by trigger, it will be or permanent loop with SELECT from table, or scheduled Job.
For scheduled Job - You do not need trigger in many cases, just need have 2 timestamp columns - CreatedAt, UpdatedAt ...
Tools and products for real CDC (I leave in list only not Talend competitors):
- MariaDB MaxScale - https://mariadb.com/products/technology/maxscale , it part of Enterprise packet, but subscription required if You use it for more than 3 server in production environment.
- Maxwell - https://github.com/zendesk/maxwell
- Apache NiFi 1.2+ https://nifi.apache.org
The first 100 community members completing the Open Studio survey win a $10 gift voucher.
Talend named a Leader.
Kickstart your first data integration and ETL projects.
Watch the recorded webinar!
Pick up some tips and tricks with Context Variables
Learn how media organizations have achieved success with Data Integration
Look at6 ways to start utilizing Machine Learning with Amazon We Services and Talend