From Thursday, July, 9, 3:00 PM Pacific,
our Community site will be in
read-only mode
through Sunday, July 12th.
Thank you for your patience.

parallel execution in iteration link : jumbles variable values

Highlighted
One Star
One Star

parallel execution in iteration link : jumbles variable values

Hi,
I tried to use the parallel execution property in iteration link, to run commands (in tSystem component) in parallel
Several commands are launched simultaneously but the parameters passed to them (via context variables and a command line) are mixed
the general schema is the following :
tMySQLInput => tFlowtoIterate =iteration link=> tJava (to treat values returned from tMySQLInput) => tSystem
For example :
the tMySQLInput returns 2 identifiers which are passed to the tSystem component
there are indeed 2 iterations but twice the same identifier is passed to the tSystem component
Have you ever noted this ?
Is there another way to do it ?
Highlighted
Community Manager

Re: parallel execution in iteration link : jumbles variable values

Hi
It sounds like a bug, which version are you using? To me, each iteration should process a different row. Can you please do a testing like this, print the value returned from tMysqlOuput on tJava, for exmaple:
tMySQLInput =row1==> tFlowtoIterate =iteration link=> tJava
on tJava:
System.out.println(row1.identifiers)
Shong
----------------------------------------------------------
Talend | Data Agility for Modern Business
Highlighted
One Star
One Star

Re: parallel execution in iteration link : jumbles variable values

Yes, I think it's a bug
I slightly modified the job to use directly the tFlowToIterate variables as you mentionned, instead of context variables (which were set in a tJava component) and it seems to work
context variables seem to be not set in the good moment in the job flow
Highlighted
Community Manager

Re: parallel execution in iteration link : jumbles variable values

Hi
Maybe there is something wrong in your code, how do you set values to context variables? Can you show us your code on tJava? Also, A screenshot of job will help us...
Shong
----------------------------------------------------------
Talend | Data Agility for Modern Business
Highlighted
Four Stars

Re: parallel execution in iteration link : jumbles variable values

HI
Let me add a simple job to your point to judge whether its i a bug or not. I have designed a simple job like
tfileinputdelimited -- main --> tFlowToInterate --iterate(x2) --> tJava --iterate(x2)-->tSystem
I have a collection of commands in tFileInputDelimited, when i running the job the output is confusing me, like it is running the same command multiple times..but without parallel execution in iteration link it is working file. Please find
the Screen shot
thanks
B. Anil Kumar
Highlighted
One Star
One Star

Re: parallel execution in iteration link : jumbles variable values

Exactly the same problem in my case : some times , the same command is executed 2 or 3 times and some commands in the list are not executed
Highlighted
One Star
One Star

Re: parallel execution in iteration link : jumbles variable values

context.code_echange = input_row.code_echange;
context.groupe_echange = input_row.groupe_echange;
context.id_execution = input_row.id_execution;
context.chemin_fichier_entree = input_row.chemin_fichier_entree;
context.adresse_mail_destinataire = input_row.adresse_mail_destinataire;
context.chemin_sortie = context.EchangesRacine + context.code_echange + "/sortie/";
context.type_envoi = input_row.type_envoi;
https://docs.google.com/file/d/0B15XMMEHVcFlSzE4MTBPd2d5RDA/edit?usp=sharing
Highlighted
Community Manager

Re: parallel execution in iteration link : jumbles variable values

Hi All
Thanks for your information, I will do a testing for this issue and come back to you, keep discussing.
Shong
----------------------------------------------------------
Talend | Data Agility for Modern Business
Highlighted
One Star
One Star

Re: parallel execution in iteration link : jumbles variable values

Hi,
any news ?
Highlighted
One Star

Re: parallel execution in iteration link : jumbles variable values

Hi Shong,
Any Update on the above discussion?
Thanks---->D
Highlighted
One Star

Re: parallel execution in iteration link : jumbles variable values

Hi,
Any update ?
Highlighted
Seventeen Stars

Re: parallel execution in iteration link : jumbles variable values

hi all,
a long time since starting post Smiley Happy
never mind ,
first be sure that you have a core for each multi-threading (or it'll be just a like-muli iteration).
second in advanced setting => extra of the job be aware to check 'multi-threading execution'
hope it hepls
regards
laurent

2019 GARTNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

Best Practices for Using Context Variables with Talend – Part 2

Part 2 of a series on Context Variables

Blog

Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables

Blog

Migrate Data from one Database to another with one Job using the Dynamic Schema

Find out how to migrate from one database to another using the Dynamic schema

Blog