Four Stars

Re-startability of jobs

I would like to implement re-startability feature in talend jobs. Basically the goal is that if a job has been executed then in the subsequent runs it should pick only those records that have been not processed in the previous run. I do not want Talend to start executing from the start again. How can I achieve this?

  • Data Integration

Re: Re-startability of jobs

The most flexible way to do this is to capture the state of your process.  I have done this before where I will have a staging/state table where I will keep the record Id, and an Action/State flag.  

For example, I can have the following state:

0 - Not processed

1 - Insert Record

2 - Update Record

3 - Update Dependencies or something else

4 - Delete Record

5 - etc etc


Then I process all the records and add/update the state table.  Then I read from the state table to do further processing.  In this way, each time I re-run the job, the job knows where it has reached in processing of records, and can re-start where it left. You can even force a record by switching the Action/State flag.