ETL - Improve speed of process

One Star

ETL - Improve speed of process

Dear friends, I have a problem in Talend Open Studio. My job is described as below: 
tOracleInput --->(1) TMap (Modify data) -->(2) tWriteJSONFields -->(3) tMongoDBOutput
In Oracle database, there are around 300k records.  The processing speed of (1) and (2) is around  3054,98rows/s, but in (3) is only around 10rows/s due to the process of tWriteJSONFields. The questions are: 
- How can I improve the speed of (3)? 
- Some time, there is an OutMemory Exception because data which is cached in (2) waiting for the process of tWriteJSONFields is over load. So can I limit a number of rows each time. More clearly, there are 300k records in Oracle, if the number of rows processed in tMap equals to an amount of rows (for example: for each 1000 records), the tOracleInput sleeps and wait for the next iteration. 
Any suggestion will be appreciate. Thanks so much!  
Moderator

Re: ETL - Improve speed of process

Hi,
For your "OutofMemory" exception, could you please take a look at KB article:TalendHelpCenter:Exception: outOfMemory to see if it works?
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.

Calling Talend Open Studio Users

The first 100 community members completing the Open Studio survey win a $10 gift voucher.

Start the survey

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

How Media Organizations Achieved Success with Data Integration

Learn how media organizations have achieved success with Data Integration

Read

Tutorial

Introduction to Talend Open Studio for Data Integration.

Watch