How to handle a large SAP table efficiently

Five Stars

How to handle a large SAP table efficiently

Task : Extract all the records from a large sap table and store in a SQL table - 39,000,000 records. 


Progress: I have created a job with a tSAPTableInput component , one tmap and a tMSSQLOutput component ( so far so good). 

 

Problem: The performance starts well but then settles into 25 records per second - this will take days to complete. 

 

Screen Shot 2019-01-23 at 12.30.49.png

 

 

Screen Shot 2019-01-23 at 12.34.24.png

Question : Is there a way to improve performance or perhaps break it into chucks effectively ? 

Employee

Re: How to handle a large SAP table efficiently

Hi,

 

    Did you try tparallelize as the starting point and from there multiple data fetch flows going to output location?

 

    Please increase the memory of the job so that Talend job can process without any hindrance.

image.png

 

     If its still taking more time, write extracted data to file in parallel mode and then use tDBBulk components instead of tDBOutput component to process data faster.

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved :-)

Five Stars

Re: How to handle a large SAP table efficiently

Dear Nikhil , 

The tDBBulk component is not available at my version. The tParallelize component did certainly speed things up but it still crashes with memory issues. 



Highlighted
Employee

Re: How to handle a large SAP table efficiently

Hi,

 

   Could you please cross check the Xms and Xmx parameters of your Talend job and fine tune them according to your system resource availability?

 

    The fine tuning of these parameters will help you to avoid crash due to memory issues. Detailed information can be fetched from below link. You can change these parameters under Advanced section of Run tab.

 

https://help.talend.com/reader/_zCTtDbUbgf9AGOJY8pz1Q/EoxO17sMiUedPquMxGLGdg

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved :-)

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

How Media Organizations Achieved Success with Data Integration

Learn how media organizations have achieved success with Data Integration

Read

Why Companies Move to the Cloud: 7 Success Stories

Learn how and why companies are moving to the Cloud

Read Now