Help me how to improve the loading performance. i have 1 lakhs record in a table and moving into destination table.
It is loading very slow. I am using Talend open studio 6.4.1
performance in this case depends from target database
You can examine settings and configuration for check how them relevant for task
also You can test job with tMySQLBulkExec instead of tMySQLOutput
I am doing insert/update option . it is not possible using tmysqloutputbulkexec.
So can you tel me how to speed up the loading process
insert/update - always slow operations, and You can not expect any chances to improve it (yes, it possible tune database, but it will be not huge difference)
You can try redesign job - there are many variant depending from data:
- You can bulk import all into database (into separate table), and use
- or update where exists + insert where not exists
- or delete where exists + insert all
P.S. it is bad practice to use local terms on international forum if You not expect answers in French or Chinese (I can also start use Russian terms ;-) ) ... but in any case 100 000 records - it is not too big for run inside database
P.P.S. of course staging table must have indexes
Introduction to Talend Open Studio for Data Integration.
Practical steps to developing your data integration strategy.
Create systems and workflow to manage clean data ingestion and data transformation.