TAC - Running parallel job using tParallelize

Seven Stars

TAC - Running parallel job using tParallelize

Hi,

 

I have designed a job to use tParallel component which has say 4 tRunJob's, where each tRunJob call the same child job.

The child job connects to a database, read the table, processed the rows using tMap, tDataShuffle and tDataMasking components, updates the same table and closes the connection.

 

This is to split the table to 8 splits and process the rows in parallel. The split is done based on the primary key of the table.

 

tParallel ----> tRunJob_1(job_process_table1 - say for 100 rows)

              ----> tRunJob_2(job_process_table1 - say for 101 to 200 rows)

              ----> tRunJob_3(job_process_table1 - say for 201 to 300 rows) 

             ----> tRunJob_4(job_process_table1 - say for 301 to 400 rows)

 

This job runs fine when run locally from Talend to the dev databases.

But fails with "Cannot establish connection error" when running from TAC.

The same job runs fine when I run with single tRunJob_1.

 

Could not understand the reason.

Could you kindly suggest what all might be the reason.

 

Thanks.

 

 

 

              

Employee

Re: TAC - Running parallel job using tParallelize

Hi,

 

    Are you trying to use the same database connection in all the child jobs?

 

    Since you are closing the connection within the child job, this could be creating some issue. Could you please try separate DB connections for your child jobs in this case and try to see whether the issue persists?

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved :-)

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables

Blog

Migrate Data from one Database to another with one Job using the Dynamic Schema

Find out how to migrate from one database to another using the Dynamic schema

Blog

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog