Very Slow Performance

One Star

Very Slow Performance

I will preface this e-mail by saying that I am brand new to Talend.
I am importing a CSV file from the metadata repository on one box to a MySQL box in the metadata repository. I have a filter on the text file. So it goes CSVInput>tFilterRow>tMap>MySQLOutput. It is extremely, extremely slow. The CSV is only ~17k rows, and the file is importing at a rate of a little less than 1 record per second. At this rate it will take more than about 5 hours. I have 6 more CSV files in exactly the same format with many, many more records. I need a way to speed this up. 17k records was nothing in DTS in MSSQL, so I must be doing something wrong. I just do not know what.
??? Any ideas where I should look. Please advise.
--Harry
One Star

Re: Very Slow Performance

use extended insert in the mysqloutput component (under advanced settings)
One Star

Re: Very Slow Performance

Take care that you do not use the debug mode in TOS. This will slow down the speed to one row a second.
Employee

Re: Very Slow Performance

Thx Volker !
Just a precision : Volker is speaking about "The trace mode"
HTH,
One Star

Re: Very Slow Performance

Thank you simony. The extended insert button was already checked.
Volker and mhirt, I had the traces button checked. When I unchecked it, that completely solved the problem. I understand the need for traces now. I just kind of figured traces was like an explain in MSSQL. I t does not make that much of a difference. Night and day. This baby really moves now. 24.61 seconds on nearly 18k rows.
I figured it was something simple and bone headed I was doing. Thanks!
--Harry

What’s New for Talend Spring ’19

Join us live for a sneak peek!

Sign up now

Agile Data lakes & Analytics

Accelerate your data lake projects with an agile approach

Watch

Definitive Guide to Data Quality

Create systems and workflow to manage clean data ingestion and data transformation.

Download

Tutorial

Introduction to Talend Open Studio for Data Integration.

Watch