General: Troubleshooting Jobs

One Star

General: Troubleshooting Jobs

Hello Community,

I'm also new to Talend. I have a question for Troubleshooting Jobs.
Some Jobs run at 30k rows/sec.
My new Job (2 million rows, over 100 Type 1 SCD Fields, several Type 2 SCD Fields) starts really good with 2k rows/sec - but it's rate is dropping pretty hard after a while. It goes down to like 150 rows/sec.


* The Job is using actively 6-7 GB of RAM
* Parameters Dfile.encoding=UTF-8 / Xms500M / Xmx8192M are set
* Before weeks, the Job did run pretty smooth! (New change was adding more fields) - But after deleting them again, no change.
* dom4j parser
* Java 1.6 
* SQL-Server where the Data is coming from: He's bored!
* SQL-Server where the Data is going in: He's bored!
* Talend version: 5.2.2 r99271-20130307-0242
* I think I have to use exactly this version, due the ERP-Systems Company approval
* I was also trying to run the Job with the .jar - no difference, slow..



So - How do you guys start Troubleshooting Jobs to increase performance?

Thank you for any ideas/informations and best regards
dit0x

One Star

Re: General: Troubleshooting Jobs

Update:
The Job runs now with 6k rows/sec - Why? I dont know! NOTHING changed.
Next time this happens, I'm going to check the network rates for reads and writes and give you an update.

15TH OCTOBER, COUNTY HALL, LONDON

Join us at the Community Lounge.

Register Now

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

How Media Organizations Achieved Success with Data Integration

Learn how media organizations have achieved success with Data Integration

Read

Definitive Guide to Data Integration

Practical steps to developing your data integration strategy.

Download