[resolved] Parallel processing of ActiveMQ data

Seven Stars

[resolved] Parallel processing of ActiveMQ data

I am trying to do some parallel processing of ActiveMQ messages. I created the following job:

I would expect the job to process 20 messages in parallel, but it keeps processing 1 message at a time.
What should I change to make parallel processing work?
-----------------------------------------------------------------------------------------------------------
When you like my answer, please accept it as solution and send some kudos

Accepted Solutions
One Star

Re: [resolved] Parallel processing of ActiveMQ data

You have an 'iterate' link from the setContext, so you're executing it one by one. In this setup you set the context (static/global) and execute the subjob using the 'iterate' link, therefore one by one.
You could use tFlowIterate component and there's a checkbox 'execute parallel'.  However - you'd overwrite the job context values and you will get into thread race condition. So you will have to pass data another way. (still there's a valid question - how?) You have to enable calling the subjobs in parallel too.
You can still use the mediation routes (in fact they are Camel routes) where you can control the parallelism using the ActiveMQ connection pool settings.
Gabriel

All Replies
Community Manager

Re: [resolved] Parallel processing of ActiveMQ data

Hi 
Do you check the 'receive number of messages' box on tMomInput component? And change the job to:
tMomInput--main--tFlowToIterate--iterate--tJava--oncomponent--tRun
Check the 'Enable parallel execution' box on iterate link.
----------------------------------------------------------
Talend | Data Agility for Modern Business
Seven Stars

Re: [resolved] Parallel processing of ActiveMQ data

When setting 'receive number of messages' to a number instead of 'keep listening' only changes the number of messages after which the jobs stops and doesn't say anything about number of parallel messages to proces.
I have created the job as you mentioned, but there is always just 1 message coming from the queue.
-----------------------------------------------------------------------------------------------------------
When you like my answer, please accept it as solution and send some kudos
One Star

Re: [resolved] Parallel processing of ActiveMQ data

You have an 'iterate' link from the setContext, so you're executing it one by one. In this setup you set the context (static/global) and execute the subjob using the 'iterate' link, therefore one by one.
You could use tFlowIterate component and there's a checkbox 'execute parallel'.  However - you'd overwrite the job context values and you will get into thread race condition. So you will have to pass data another way. (still there's a valid question - how?) You have to enable calling the subjobs in parallel too.
You can still use the mediation routes (in fact they are Camel routes) where you can control the parallelism using the ActiveMQ connection pool settings.
Gabriel
Seven Stars

Re: [resolved] Parallel processing of ActiveMQ data

Thanks, I already had some fear for thread race condition. This will probably also appear with hashes used. 
-----------------------------------------------------------------------------------------------------------
When you like my answer, please accept it as solution and send some kudos