How to run the ETL job automatically when trigger Fires (CDC)

Highlighted
Seven Stars
Seven Stars

How to run the ETL job automatically when trigger Fires (CDC)

Hi Team ,

I would like to design a Talend Job which will trigger automatically whenever there is a change in the tables(Inserts/Updates) and process those changes to corresponding databases. I am using Talend Open Studio , Mysql - DB's. Would it be possible to trigger the talend Job only there is a change occured in the tables ?

 

Thanks,

hmd.

 

Highlighted
Community Manager

Re: How to run the ETL job automatically when trigger Fires (CDC)

Hello
The CDC feature is only available in Talend Enterprise Subscription product, for more information about CDC, please read this page:
https://help.talend.com/reader/sSAk2HQITuEIWJdkl3e1Rw/xMY3zAjqxlBYJS7sin671w

Regards
Shong
----------------------------------------------------------
Talend | Data Agility for Modern Business
Highlighted
Forteen Stars

Re: How to run the ETL job automatically when trigger Fires (CDC)

Shong!

For proper CDC - use 3rd party tools for capture CDC from MySQL to Kafka and then Talend Listen Kafka and parse changes

 

Trigger Based CDC can be created with OpenSource version as well - add triggers, read the table + realise logic for multi subscribers, it as well allow handle situations when same records was - INSERTED/UPDATED and than DELETED from table. 

All problem of any trigger based solution - it not fire Job by trigger, it will be or permanent loop with SELECT from table, or scheduled Job.

 

For scheduled Job - You do not need trigger in many cases, just need have 2 timestamp columns - CreatedAt, UpdatedAt ... 

Tools and products for real CDC (I leave in list only not Talend competitors):
- MariaDB MaxScale - https://mariadb.com/products/technology/maxscale , it part of Enterprise packet, but subscription required if You use it for more than 3 server in production environment.

- Maxwell - https://github.com/zendesk/maxwell

-  http://debezium.io/docs/connectors/mysql/

- Apache NiFi 1.2+ https://nifi.apache.org

 

 

-----------

2019 GARTNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables

Blog

Migrate Data from one Database to another with one Job using the Dynamic Schema

Find out how to migrate from one database to another using the Dynamic schema

Blog

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog