Data insertion performance using Talend DI Jobs

Four Stars

Data insertion performance using Talend DI Jobs

Talend version - 6.1.1. 
SVN Version - svn, version 1.6.11 (r934486) 
CentOS release 6.7 (Final) 
DB - MySQL 5.7.13 InnoDB Engine
Infrastructure hosted in Azure Cloud
Environment - Pre-PROD

We are facing performance issues while inserting the data into MySQL DB using the Talend jobs deployed in TAC.
Problem Statement :
# Job design - with only insert in tmySQLoutput component, data flow rate is >2000rows/sec.
# Job design - with only update in tmySQLoutput component, data flow rate is >2000rows/sec
# Job design - with insert or update in tmySQLoutput component, data flow rate is very poor. approx 3-4rows/sec.

We are having the same architecture in development environment. in DEV env, even jobs with design as insert or update in tmySQLoutput component, data flow is far better >1000 rows/sec.

Talend DI Server & MySQL DB versions are same in both the environments. But facing the data flow issue in Pre-PROD env.

Can anyone provide help on this issue.

Thanks
Ashok Karnam

Thanks & Regards
Ashok Karnam
Moderator

Re: Data insertion performance using Talend DI Jobs

Hi,
What does your job design look like? Have you already checked out "Extend Insert " option on tMySqlOutput component to carry out a bulk insert of a defined set of lines?
 Have you already reported a ticket on Talend Support Portal for your subscription version? In this way, we can give you a remote assistance through support cycle with priority.
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.

15TH OCTOBER, COUNTY HALL, LONDON

Join us at the Community Lounge.

Register Now

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

How Media Organizations Achieved Success with Data Integration

Learn how media organizations have achieved success with Data Integration

Read

6 Ways to Start Utilizing Machine Learning with Amazon We Services and Talend

Look at6 ways to start utilizing Machine Learning with Amazon We Services and Talend

Blog