Multi threading in talend open studio

One Star

Multi threading in talend open studio

How can we achieve multi threading using talend for open studio ? any ideas would help.
Thanks,
Chandra
Seventeen Stars

Re: Multi threading in talend open studio

I use actually only one method which has been proven to be stable and scalable: Parallel execution of the Iterate trigger.
My job design is mostly separated in 2 jobs:
First job which proceed a piece of work (file, country, day etc.) and I call this job a worker job.
Second job detects the work to be done and iterate (enable here the parallelisation in the advanced settings of the Iterator) through the work items and use tRunJob to call the worker. The worker gets its work by context variables.
This has a lot of advantages:
1. you can easily test the worker (set for all necessary input context variables a prompt)
2. the amount of running worker job will be kept constantly by the engine
3. the work progress can easily be monitored
4. if you use the option independent child job in the tRunJob component you avoid memory leaks
5. This design works also in the Open Studio.

Calling Talend Open Studio Users

The first 100 community members completing the Open Studio survey win a $10 gift voucher.

Start the survey

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

How Media Organizations Achieved Success with Data Integration

Learn how media organizations have achieved success with Data Integration

Read

Definitive Guide to Data Quality

Create systems and workflow to manage clean data ingestion and data transformation.

Download