Parallel processing

Five Stars

Parallel processing

Hi, 

I am using the TOS big data 6.4 i have a job that contains the an excel sheet which contains approximately 10000 rows. i want to create a job where all my 10000 rows need to establish 10000 connections with the data base and want to insert into database parallel. That means i want to create 10000 connections how should i do this with the talend open studio...please help me...

 

Thanks & Regards

Mastan Shaik 

Community Manager

Re: Parallel processing

From what you have said, it sounds like you want to do this on a single a machine and not in a Hadoop node cluster. If this is the case you need to think about what you are asking. Parallel processing is great, but it depends on how many cores you have available. If your machine has 10,000 cores, then you are in luck. Otherwise, you will probably end up massively slowing the processing of your data. When running parallel processing you should start by using the rule n-1 processes where n is the number of cores

Five Stars

Re: Parallel processing

Thanks for your reply,

Here i want to check the data base capability for that i need to create approximately 10000 connections through talend from a single machine. i have a excel sheet having 10000 rows i want to insert all the rows parallel by creating that many number of connections to check my data base is stable or not. this is my requirement.

 

 

Thanks & Regards 

Community Manager

Re: Parallel processing

You are not going to be able to do this on one machine. As I said in my last post, a rule of thumb for parallel processing is number of cores-1  = number of parallel processes. Unless you have 10000 cores, you won't be able to achieve this on one machine

Calling Talend Open Studio Users

The first 100 community members completing the Open Studio survey win a $10 gift voucher.

Start the survey

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

How Media Organizations Achieved Success with Data Integration

Learn how media organizations have achieved success with Data Integration

Read

6 Ways to Start Utilizing Machine Learning with Amazon We Services and Talend

Look at6 ways to start utilizing Machine Learning with Amazon We Services and Talend

Blog