I am using the TOS big data 6.4 i have a job that contains the an excel sheet which contains approximately 10000 rows. i want to create a job where all my 10000 rows need to establish 10000 connections with the data base and want to insert into database parallel. That means i want to create 10000 connections how should i do this with the talend open studio...please help me...
Thanks & Regards
From what you have said, it sounds like you want to do this on a single a machine and not in a Hadoop node cluster. If this is the case you need to think about what you are asking. Parallel processing is great, but it depends on how many cores you have available. If your machine has 10,000 cores, then you are in luck. Otherwise, you will probably end up massively slowing the processing of your data. When running parallel processing you should start by using the rule n-1 processes where n is the number of cores.
Thanks for your reply,
Here i want to check the data base capability for that i need to create approximately 10000 connections through talend from a single machine. i have a excel sheet having 10000 rows i want to insert all the rows parallel by creating that many number of connections to check my data base is stable or not. this is my requirement.
Thanks & Regards
You are not going to be able to do this on one machine. As I said in my last post, a rule of thumb for parallel processing is number of cores-1 = number of parallel processes. Unless you have 10000 cores, you won't be able to achieve this on one machine
The first 100 community members completing the Open Studio survey win a $10 gift voucher.
Talend named a Leader.
Kickstart your first data integration and ETL projects.
Watch the recorded webinar!
Pick up some tips and tricks with Context Variables
Learn how media organizations have achieved success with Data Integration
Look at6 ways to start utilizing Machine Learning with Amazon We Services and Talend