want to run all pipelines parlally

One Star

want to run all pipelines parlally

Hi All..
i have 3000 records in my source. and i have three pipelines in my job.
My requirement is first pipeline needs to send first 1000 records to target and second pipeline needs to send the next 1000 records to target and third pipeline needs to send next 1000 records to target.
the source and target for the three pipelines are same.

I tried in this way and i got it
in the first source component i defined as Header=0 and footer=2000
and in the second source component Header=1000 and footer=1000
and in the third source component Header=2000 and footer=0
Is it proper way or There is another way to get this ?
I want to run all pipelines parlally.
Community Manager

Re: want to run all pipelines parlally

Other ways is to use tMap or tSampleRow to split the rows into three data flow, for example, below is an example job with tSampleRow component.
--main--tSampleRow_2--->tLogRow _2
on tSampleRow_1:
set the range as "1..1000"
on tSampleRow_2:
set the range as "1001..2000"
on tSampleRow_3:
set the range as "2001..3000"
This job only contains one subjob, it does not run parallel.
There has three subjobs in your job design, if you want to run all pipelines parallel, open Job settings-->Extra tab, check the option 'Multi thread execution'.
Talend | Data Agility for Modern Business

What’s New for Talend Spring ’19

Watch the recorded webinar!

Watch Now


Introduction to Talend Open Studio for Data Integration.


Downloads and Trials

Test drive Talend's enterprise products.


Definitive Guide to Data Integration

Practical steps to developing your data integration strategy.