Can you run jobs in parallel using tRunJob set to dynamic?

One Star

Can you run jobs in parallel using tRunJob set to dynamic?

Hi
I have now got a job set up to dynamically run other jobs passed in from a database table. I.e. I can decide which say 30 jobs to run out of say 50 jobs. But I want the jobs to run in parallel for performance reasons. At the moment the tRunJob waits for each job to finish before starting the next. Is there any way around this? I have the main job (the one with the tRunJob component in it) set to multi threaded and have tried checking and un checking the "die on child error" option on the tRunJob. If what I am trying to do will not work with tRunJob is there any other way to do it?
This is the structure of the main job
tOracleInput (returns the list of jobs to run)
|
tFlowToIterate (sets a key to the job name)
|
tRunJob (Set to dynamic with the Context job field set to the key set above)
All of the jobs are in the Job list
Many thanks in advance
Dave

Re: Can you run jobs in parallel using tRunJob set to dynamic?

Hello david,
i have created something similar to what you are trying to do.
running jobs in parallel - but instead of calling them from a tRunJob, i have set the 'child' job as a web-service and deployed on multiple hosts. passing parameter on the web-services and checking for their results in a DB table.
does that help?
Nicolas
One Star

Re: Can you run jobs in parallel using tRunJob set to dynamic?

Hi Nicolas
Many thanks for the response. I have not looked at the web service approach so I will take a look at this. We currently deploy all of our jobs as Java executables on Unix for scheduling with an enterprise scheduling tool, so the use of web services may be a problem.
I also meant to add to my post. Is there a difference between how tRunJob is implemented in TOS 4.2 and TIS 4.2? I am using TOS 4.2 while waiting for the TIS 4.2 install. I notice that there is no tparrallisze component in TOS 4.2.
Dave

Re: Can you run jobs in parallel using tRunJob set to dynamic?

there are differences between the TIS and TOS version, and rightly so - since you are paying for the greater options on TIS.
see here:
http://www.talend.com/products-data-integration/matrix.php

Re: Can you run jobs in parallel using tRunJob set to dynamic?

daveseavista, With TIS, you can parallelize your iterate connectors and this will probably work with the dynamic tRunJob-- but I haven't tried it so please let us know how it works out.
Six Stars

Re: Can you run jobs in parallel using tRunJob set to dynamic?

Hi Nicolas, 

 

Could you please let us know how to set the child job as a web service or post a pic of the job if possible or any blog ? I have googled and couldn't find any documentation. 

 

Thanks in advance

 

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 2

Part 2 of a series on Context Variables

Blog

Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables

Blog

Migrate Data from one Database to another with one Job using the Dynamic Schema

Find out how to migrate from one database to another using the Dynamic schema

Blog