Job running very very slowly

Four Stars

Job running very very slowly

Hi Talend Experts,

 

My job is running very very slow (throughput is 45 rows per/Sec) / Total number of records 212960 - from 7 files. I'm scratching my head to find out the bottlenecks, but no luck Smiley Sad.

 

For example. Below is the order of the components. I have attached two screen shots. 1. Actual Job flow. 2. tMap

 

tFileList --> tfileInputDelimited --> tmap --> tOracleOutput --> tfileOutputDelimited (Capture Rejects)

 

Below are the optimization techniques i have done so far.

1) In the Run tab, Advanced setting, checked the Use specific JVM arguments and changed the values to

    -Xms1024M

   - Xmx8048M

 

2) Changed the values in the Ini file in bold below (Talend setup location)

    -vm
C:\Program Files\Java\jdk1.8.0_202\jre\bin
-vmargs
-Xms2048m
-Xmx8196
-Dfile.encoding=UTF-8
-Dosgi.requiredJavaVersion=1.8
-XX:+UseG1GC
-XX:+UseStringDeduplication

 

Still its slow.. Kindly let me know what else i need to do to improve the performance.

 

Thank you

Kumar

Highlighted
Employee

Re: Job running very very slowly

Hi,

 

    Performance tuning is a very broad subject but lets try some basic tuning exercises.

 

image.png

 

Could you please increase the batch size first and see the results? Also try parallel execution.

 

In the meantime, could you please check the network bandwidth between your machine and Oracle Database? Do you have enough capacity in sending network packets?

 

Another item to check is the overall performance of Oracle DB. Could you please check with Oracle DBA whether the DB is having good performance matrices at DB level before and during data load. This will give you clue in case the issue is due to DB side problems.

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved :-)


Warm Regards,
Nikhil Thampi
Please appreciate our members by giving Kudos for spending their time for your query. If your query is answered, please mark the topic as resolved :-)
Four Stars

Re: Job running very very slowly

I Appreciate your rapid response Nikhil.

 

Below are my responses to your suggestions.

1)Could you please increase the batch size first and see the results? Also try parallel execution.

             Increased batch size to 100000. still throughput is 45 rows/ Sec Smiley Sad.  Don't see Parallel execution option under tStatcatcherStatistics. Where do i find to

             enable it?

2) In the meantime, could you please check the network bandwidth between your machine and Oracle Database? Do you have enough capacity in sending network packets?

             May i please know how do i check the network bandwidth btw my machine and Oracle Dbase?

 

3) Another item to check is the overall performance of Oracle DB. Could you please check with Oracle DBA whether the DB is having good performance matrices at

     DB level before and during data load. This will give you clue in case the issue is due to DB side problems.

           Sure. Will do.

 

 

Kindly advise. 

 

Kumar

 

Thank you.

Forteen Stars

Re: Job running very very slowly

@kumara 

 

100000 - it is too much, start with 10000 as suggested by @nikhilthampi 

 

do you just insert or insert/update?

 

-----------
Employee

Re: Job running very very slowly

@kumara 

 

The best team to advise about network bandwidth will be your Network Administrators. Could you please contact them and I am sure they will have nice graphs and reports about the network performance. 

 

@vapukov 's point is also very important one. You need to see the number of inserts vs updates. Once you get more infor from DBAs, you will get more clarity about the tuning opportunities at DB level.

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved :-)


Warm Regards,
Nikhil Thampi
Please appreciate our members by giving Kudos for spending their time for your query. If your query is answered, please mark the topic as resolved :-)
Forteen Stars

Re: Job running very very slowly

@nikhilthampi 

 

yes, of course!

 

but more - not depending from the number of Inserts vs Updates (even if 100% rows new) with settings "INSERT or UPDATE", the component works up to 100 times slower than just insert

it is because of a different form of queries used for insert data

-----------

What’s New for Talend Spring ’19

Watch the recorded webinar!

Watch Now

Definitive Guide to Data Quality

Create systems and workflow to manage clean data ingestion and data transformation.

Download

Tutorial

Introduction to Talend Open Studio for Data Integration.

Watch

Downloads and Trials

Test drive Talend's enterprise products.

Downloads