Multithread, iterate and tFlowIterate

Four Stars

Multithread, iterate and tFlowIterate

Hello

 

I am using Talend TOS (V7), I don't have access to Enterprise version, unluckily

 

I have a job with the following design (see the attached screenshot)

 

First a query that select among the other info, n couples colorID/sizeID

Then a tFlowToIterate to make colorID and sizeID available in an iteration

On component OK the colorID/sizeID are used as filters to retrieve the data from an Oracle Table

 

The job works fine, but when the number of colorId/SizeId are many, it is quite slow to complete.

I have tried enabling iterate parallel parameters and things run much better (the execution time halves, of course).

The problem is that reading the documentation I have found that globalMap is not thread safe so I feel there are risks in making my design multithread (a mix up of colorIds or sizeIds during the executions would cause serious problems)

 

Is there a way to modify my design in order to make it thread safe?

 

Thanks for your help

Alessandro

 

Basicall

 


Accepted Solutions
Twelve Stars

Re: Multithread, iterate and tFlowIterate

Capture.PNG

 

yes, you may iterate to trunjob using context value in place of global.
with that configuration thread have there own variables.

be careful tjava is not explicite so we do not know what is the objectif of your job. not easy for maintenance.

Francois Denis

Tag as "solved" for others! Kudos to thanks!


All Replies
Twelve Stars

Re: Multithread, iterate and tFlowIterate

Capture.PNG

 

yes, you may iterate to trunjob using context value in place of global.
with that configuration thread have there own variables.

be careful tjava is not explicite so we do not know what is the objectif of your job. not easy for maintenance.

Francois Denis

Tag as "solved" for others! Kudos to thanks!

Four Stars

Re: Multithread, iterate and tFlowIterate

The tJava does some simple log, no biggies Smiley Happy

 

So if I understand well I

- keep the iterate with parallel execution enabled

- use the globalMap values JUST to populate subjob context Parameters

 

Should I check the "use an independent process to run subjob" too?

 

Many thanks for your help

Alessandro

 

Twelve Stars

Re: Multithread, iterate and tFlowIterate

please to help.
validate the solution on your side and check as solved.
By

Francois Denis

Tag as "solved" for others! Kudos to thanks!

Four Stars

Re: Multithread, iterate and tFlowIterate

Hello

I will for sure, as soon as I tested...

Just a detail : Should I check the "use an independent process to run subjob" too or the check in the iterate flow is enough?

Twelve Stars

Re: Multithread, iterate and tFlowIterate

yes check it then you will have independent java thread.

Francois Denis

Tag as "solved" for others! Kudos to thanks!

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 2

Part 2 of a series on Context Variables

Blog

Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables

Blog

Best Practices for Using Context Variables with Talend – Part 3

Read about some useful Context Variable ideas

Blog