One Star

Job Admin Tool (TIS only) - Run sequential series of jobs?

It is possinble to use the job administration tool (available in TIS only), is it possible to run a series of jobs that are (logically) linked together to run sequentially (one after another), such that if one fails, any subsequent jobs in the series do NOT run?
I am trying to architect separate and discrete ETL jobs, that have no dependencies between them, other than they should run one after another in succession, and that the next job only runs if the previous job succeeds.
While I realize that I can create one job that calls other jobs within it, this is not bext practice ETL, in my opinion.

Dave
9 REPLIES
Six Stars

Re: Job Admin Tool (TIS only) - Run sequential series of jobs?

In others systems this good principle is usually implemented by the means of a job queue ( the job queue is decoupled by the scheduler, you just submit ETL jobs to the queue and it runs them sequentially, no need to set precise scheduled time).
It is a very useful concept ( obvious, necessary and built-in in mainframes and as400 operating systems ) to ensure data consistency, enforce strict sequence of jobs, control load.
I found myself in many windows schedulers stuck by the need of a sequential job queue (and at the end you end up setting precise time schedules just to avoid conflicts ).
I was considering to evaluate TIS... does the TIS scheduler have the concept of a "job queue"?
Would be good to say: give a set of ETL jobs, submit them to a job queue ( the system will execute them one by one ).
One Star

Re: Job Admin Tool (TIS only) - Run sequential series of jobs?

Nor sure about th ejob queue - new to TIS.
I was hoping to be able to schedule a group of job, to run in sequence. I am used to scheduling and running Jobs in SQL server's SQL agent, where a job can have steps, run in succession, in which each step may be an ETL job. So, if a step fails, the job fails (good).
That way, the job steps are decoupled from each other in terms of coding dependencies which would otherwise have to be introduced by running a Talend "Master" Job, that calls other jobs.
Six Stars

Re: Job Admin Tool (TIS only) - Run sequential series of jobs?

In my opinion a feature like that is mandatory in enterprise software ( not in TOS but of course must be in TIS to consider it "enterprise" grade ).
Four Stars

Re: Job Admin Tool (TIS only) - Run sequential series of jobs?

Hello,
Currently, it's possible to make a "Master" job for scheduling other jobs same as "job queue".
However, we had alreay an internal disccusion at talend for adding orchestration in the Job Conductor of TIS. This functionnality is in our roadmap, maybe for 3.3 version.
Can you declare the feature on our bugtrack? It's always interesting to have some feedback of our users on new functionnality.

Regards,
SeB
Six Stars

Re: Job Admin Tool (TIS only) - Run sequential series of jobs?

Smalleret, master job concept introduce code dependencing between the jobs, and you force at code layer what should be delegated at system layer.
In a standard job queue each job is independent and not necessary related.
thanks
One Star

Re: Job Admin Tool (TIS only) - Run sequential series of jobs?

Hello,
Currently, it's possible to make a "Master" job for scheduling other jobs same as "job queue".
However, we had alreay an internal disccusion at talend for adding orchestration in the Job Conductor of TIS. This functionnality is in our roadmap, maybe for 3.3 version.
Can you declare the feature on our bugtrack? It's always interesting to have some feedback of our users on new functionnality.

Regards,
SeB

Sure - I will to that. That functionality would then give Talend the same level of job control as SSIS.
One Star

Re: Job Admin Tool (TIS only) - Run sequential series of jobs?

If you have your first job create a dummy output file on completion with the tFileTouch, you can set up a file trigger in job conductor for job2 to trigger when this file is created. Then delete it at the end of job2. This can be repeated for more jobs in the sequence.
One Star

Re: Job Admin Tool (TIS only) - Run sequential series of jobs?

Hi
Is there any update on this? We have in the mean time several TIS projects which are dependend on each other. So we can put them in 1 master job as I understand (we do this within a project already).
Regards
Petr
One Star

Re: Job Admin Tool (TIS only) - Run sequential series of jobs?

See above!