Very Slow Performance

One Star

Very Slow Performance

I will preface this e-mail by saying that I am brand new to Talend.
I am importing a CSV file from the metadata repository on one box to a MySQL box in the metadata repository. I have a filter on the text file. So it goes CSVInput>tFilterRow>tMap>MySQLOutput. It is extremely, extremely slow. The CSV is only ~17k rows, and the file is importing at a rate of a little less than 1 record per second. At this rate it will take more than about 5 hours. I have 6 more CSV files in exactly the same format with many, many more records. I need a way to speed this up. 17k records was nothing in DTS in MSSQL, so I must be doing something wrong. I just do not know what.
??? Any ideas where I should look. Please advise.
--Harry
One Star

Re: Very Slow Performance

use extended insert in the mysqloutput component (under advanced settings)
One Star

Re: Very Slow Performance

Take care that you do not use the debug mode in TOS. This will slow down the speed to one row a second.
Employee

Re: Very Slow Performance

Thx Volker !
Just a precision : Volker is speaking about "The trace mode"
HTH,
One Star

Re: Very Slow Performance

Thank you simony. The extended insert button was already checked.
Volker and mhirt, I had the traces button checked. When I unchecked it, that completely solved the problem. I understand the need for traces now. I just kind of figured traces was like an explain in MSSQL. I t does not make that much of a difference. Night and day. This baby really moves now. 24.61 seconds on nearly 18k rows.
I figured it was something simple and bone headed I was doing. Thanks!
--Harry

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

How Media Organizations Achieved Success with Data Integration

Learn how media organizations have achieved success with Data Integration

Read

Downloads and Trials

Test drive Talend's enterprise products.

Downloads