One Star

[resolved] Prevent running multiple instances of a Talend job

Sometimes a Talend job needs to be executed on a tight schedule, every few minutes for example. Occasionally it is undesirable to have multiple instances of the job run in parallel. This might happen in case a run takes longer than usual, and the subsequent run starts before the current one finishes. What are possible ways to prevent running multiple instances of a scheduled Talend job?
1 ACCEPTED SOLUTION

Accepted Solutions
Five Stars

Re: [resolved] Prevent running multiple instances of a Talend job

A simple method if you do not have a scheduler to handle this is to create a lock file when you start, and remove it when you finish.
Each time you start, check to see if the lock file exists.
If it exists, this tells you that the Job is either running, or has failed and has not cleaned-up.
If you handle your errors correctly, you can delete the lock file in a 'handled' error, allowing the next iteration to pick things up.
5 REPLIES
Five Stars

Re: [resolved] Prevent running multiple instances of a Talend job

Do you ever want the job to run in parallel?  If not, just place the call to the job in a managed iteration that will insure that the job is only called once at a time. If so, just add a trigger (to dictate that the job is running) that is evaluated before the job is called. 
There are no doubt more ways to accomplish what you are looking for but these two may work for you.    

Re: [resolved] Prevent running multiple instances of a Talend job

1) if you're using a enterprise scheduler (autosys/jobscheduler/batchman  etc... ) check the docs as almost all of them support exclusive exectuion.
2) as tgrady suggests, using a sentinel ( a file would be the simplest ) that gets created on job start and deleted on job end would be another way. 
One Star

Re: [resolved] Prevent running multiple instances of a Talend job

You may want to use Talend Admin center (comes with paid subscription) where the tasks created within will never execute if the first execution hasn't completed yet. This is very simple web based application to schedule job executions, monitor the progress and recover in case of failure. In general TAC provides many other features to enable virtual servers for job executions etc.
If you are using open studio and do not have TAC, try to create a database control table to capture start and end of execution. Based on the status on the control table, you can direct the execution of actual data load.
Hope it helps
Thanks 
Rohan Suresh 
Five Stars

Re: [resolved] Prevent running multiple instances of a Talend job

A simple method if you do not have a scheduler to handle this is to create a lock file when you start, and remove it when you finish.
Each time you start, check to see if the lock file exists.
If it exists, this tells you that the Job is either running, or has failed and has not cleaned-up.
If you handle your errors correctly, you can delete the lock file in a 'handled' error, allowing the next iteration to pick things up.
One Star

Re: [resolved] Prevent running multiple instances of a Talend job

Great ideas, thank you.