Jobs running slow

One Star

Jobs running slow

Hie
Could someone help me please: I have a job with an input delimited file which has about 500 000 rows could be more in future, I designed a simple job with so many lookup tables and processes using TlogRow and the Tmap being the main one. This job is taking long to run, it is taking more than a day or two to run.
Could someone tell me the best way/components to use for a job that will consists of inputdelimited csv files, MySql Tables to run faster.
I cant seem to attach the screen shot right now, if possible i can email you, if you think you can help
Many Thanks
One Star

Re: Jobs running slow

There are various approaches you might take but you need to check where it is actually bottlenecking right now. Do the lookup tables have many differnt values or just a few? Are they cached? Indexed etc? If your mysql backend is on a more powerful system then you might stage the input data and then use a join (outer if necessary) to do the lookups and produce a new data stream to feed the rest of the job.
T
Moderator

Re: Jobs running slow

Hi,
Get ride of tLogRow, it is a debugging component to trace on console.
You should only use it on small amount od data in order to test (<10000 rows)
tLogRow will slow down the whole data flow to around 100row/sec...
benjamin

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 2

Part 2 of a series on Context Variables

Blog

Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables

Blog

Migrate Data from one Database to another with one Job using the Dynamic Schema

Find out how to migrate from one database to another using the Dynamic schema

Blog