Automate to kick-start the job

Eight Stars

Automate to kick-start the job

Here's my job

 

MSSQLInputTable-->tMap-->OutputExcel

 

Whenever there's any update or a new entry happened in the source table then the above job should kick start and send it to excel. Please suggest

 

Sabrina suggested CDC feature, but it looks like the free version is not having this CDC feature available. 

 

Is there any alternative?


Accepted Solutions
Highlighted
Community Manager

Re: Automate to kick-start the job

There are a couple of things you can do here. It depends on whether you need this to be updated in (near to) real time or in batch several times a day.

 

For both solutions you could create an extra table in your database. Set up triggers on the table you are updating. The triggers should write to the new table you have created to indicate the type of action (insert, update, delete) and what has changed. This is essentially how the Talend CDC functionality works.

 

If you want a batch system (much easier), just write a Talend job to monitor this new table and carry over any changes to the other system in order they are logged in that table. Once the changes are carried over, delete them in the log table Pretty straight forward.

 

Now for a (near to) real time solution, you will need to use a Talend Route (actually an Apache Camel Route) with the Apache Camel SQL component (http://camel.apache.org/sql-component.html). Use the Route to scan for changes in the new table and if any are found, run the job described in the batch solution (or a variant of that). This will be quite tricky if you have never done this before and will require that you set up the Talend ESB system.

 

 


All Replies
Highlighted
Community Manager

Re: Automate to kick-start the job

There are a couple of things you can do here. It depends on whether you need this to be updated in (near to) real time or in batch several times a day.

 

For both solutions you could create an extra table in your database. Set up triggers on the table you are updating. The triggers should write to the new table you have created to indicate the type of action (insert, update, delete) and what has changed. This is essentially how the Talend CDC functionality works.

 

If you want a batch system (much easier), just write a Talend job to monitor this new table and carry over any changes to the other system in order they are logged in that table. Once the changes are carried over, delete them in the log table Pretty straight forward.

 

Now for a (near to) real time solution, you will need to use a Talend Route (actually an Apache Camel Route) with the Apache Camel SQL component (http://camel.apache.org/sql-component.html). Use the Route to scan for changes in the new table and if any are found, run the job described in the batch solution (or a variant of that). This will be quite tricky if you have never done this before and will require that you set up the Talend ESB system.

 

 

Eight Stars

Re: Automate to kick-start the job

Thanks for the tips rhall.

 

I have not tried this before, may require more time for the analysis and then think about the solution. Your tips will really help in this.

Community Manager

Re: Automate to kick-start the job

To make this as efficient as possible you will need to understand all of the requirements. What I described is a thorough method, but is quite processor intensive.

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

How Media Organizations Achieved Success with Data Integration

Learn how media organizations have achieved success with Data Integration

Read

Why Companies Move to the Cloud: 7 Success Stories

Learn how and why companies are moving to the Cloud

Read Now