The Definitive Guide to Data Quality
How to trigger a Talend job based on the mysql insertion/updation on a particular table.And Talend job will fetch those new/updated records and sync those into another database.
Can any once please suggest me on this.
Could you please take a look at talend CDC feature TalendHelpCenter:CDC+architectural+overview to see if it is satisfying your needs?
Note: CDC feature is only available in Talend Enterprise Subscription Version(we have provided community user with a free trial of Talend Enterprise Data Integration: http://www.talend.com/enterprise-data-integration).
For TOS workaround could be - Assume you are using some query at source to identify new insertion and another query to find updates to existing database table.
Use tReplicate component to replicate the insertion which will give another flow and use your second job/subjob within the job to insert data into the another database.
Carry same process for Update as well.
As matter of fact, talend CDC architecture is based on a publisher/subscriber model. The publisher captures the data changes and makes them available to the subscribers (Talend Jobs). Subscribers utilize the data changes obtained from the publisher.
It is a trigger or a DB log depends on what DB you are using, it closes to 100% real time. The changed data can be recorded in table when you take some action on you DB.
There is a sign-bit in subscriber table which will be set to 0 after reading record.
chmod +x filename.sh