General: Troubleshooting Jobs

Highlighted
One Star

General: Troubleshooting Jobs

Hello Community,

I'm also new to Talend. I have a question for Troubleshooting Jobs.
Some Jobs run at 30k rows/sec.
My new Job (2 million rows, over 100 Type 1 SCD Fields, several Type 2 SCD Fields) starts really good with 2k rows/sec - but it's rate is dropping pretty hard after a while. It goes down to like 150 rows/sec.


* The Job is using actively 6-7 GB of RAM
* Parameters Dfile.encoding=UTF-8 / Xms500M / Xmx8192M are set
* Before weeks, the Job did run pretty smooth! (New change was adding more fields) - But after deleting them again, no change.
* dom4j parser
* Java 1.6 
* SQL-Server where the Data is coming from: He's bored!
* SQL-Server where the Data is going in: He's bored!
* Talend version: 5.2.2 r99271-20130307-0242
* I think I have to use exactly this version, due the ERP-Systems Company approval
* I was also trying to run the Job with the .jar - no difference, slow..



So - How do you guys start Troubleshooting Jobs to increase performance?

Thank you for any ideas/informations and best regards
dit0x

Highlighted
One Star

Re: General: Troubleshooting Jobs

Update:
The Job runs now with 6k rows/sec - Why? I dont know! NOTHING changed.
Next time this happens, I'm going to check the network rates for reads and writes and give you an update.

2019 GARTNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

Best Practices for Using Context Variables with Talend – Part 2

Part 2 of a series on Context Variables

Blog

Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables

Blog

Migrate Data from one Database to another with one Job using the Dynamic Schema

Find out how to migrate from one database to another using the Dynamic schema

Blog