Seven Stars

Automate to kick-start the job

Here's my job

 

MSSQLInputTable-->tMap-->OutputExcel

 

Whenever there's any update or a new entry happened in the source table then the above job should kick start and send it to excel. Please suggest

 

Sabrina suggested CDC feature, but it looks like the free version is not having this CDC feature available. 

 

Is there any alternative?

1 ACCEPTED SOLUTION

Accepted Solutions
Fifteen Stars

Re: Automate to kick-start the job

There are a couple of things you can do here. It depends on whether you need this to be updated in (near to) real time or in batch several times a day.

 

For both solutions you could create an extra table in your database. Set up triggers on the table you are updating. The triggers should write to the new table you have created to indicate the type of action (insert, update, delete) and what has changed. This is essentially how the Talend CDC functionality works.

 

If you want a batch system (much easier), just write a Talend job to monitor this new table and carry over any changes to the other system in order they are logged in that table. Once the changes are carried over, delete them in the log table Pretty straight forward.

 

Now for a (near to) real time solution, you will need to use a Talend Route (actually an Apache Camel Route) with the Apache Camel SQL component (http://camel.apache.org/sql-component.html). Use the Route to scan for changes in the new table and if any are found, run the job described in the batch solution (or a variant of that). This will be quite tricky if you have never done this before and will require that you set up the Talend ESB system.

 

 

Rilhia Solutions
3 REPLIES
Fifteen Stars

Re: Automate to kick-start the job

There are a couple of things you can do here. It depends on whether you need this to be updated in (near to) real time or in batch several times a day.

 

For both solutions you could create an extra table in your database. Set up triggers on the table you are updating. The triggers should write to the new table you have created to indicate the type of action (insert, update, delete) and what has changed. This is essentially how the Talend CDC functionality works.

 

If you want a batch system (much easier), just write a Talend job to monitor this new table and carry over any changes to the other system in order they are logged in that table. Once the changes are carried over, delete them in the log table Pretty straight forward.

 

Now for a (near to) real time solution, you will need to use a Talend Route (actually an Apache Camel Route) with the Apache Camel SQL component (http://camel.apache.org/sql-component.html). Use the Route to scan for changes in the new table and if any are found, run the job described in the batch solution (or a variant of that). This will be quite tricky if you have never done this before and will require that you set up the Talend ESB system.

 

 

Rilhia Solutions
Seven Stars

Re: Automate to kick-start the job

Thanks for the tips rhall.

 

I have not tried this before, may require more time for the analysis and then think about the solution. Your tips will really help in this.

Fifteen Stars

Re: Automate to kick-start the job

To make this as efficient as possible you will need to understand all of the requirements. What I described is a thorough method, but is quite processor intensive.

Rilhia Solutions