ETL - Improve speed of process

Highlighted
One Star

ETL - Improve speed of process

Dear friends, I have a problem in Talend Open Studio. My job is described as below: 
tOracleInput --->(1) TMap (Modify data) -->(2) tWriteJSONFields -->(3) tMongoDBOutput
In Oracle database, there are around 300k records.  The processing speed of (1) and (2) is around  3054,98rows/s, but in (3) is only around 10rows/s due to the process of tWriteJSONFields. The questions are: 
- How can I improve the speed of (3)? 
- Some time, there is an OutMemory Exception because data which is cached in (2) waiting for the process of tWriteJSONFields is over load. So can I limit a number of rows each time. More clearly, there are 300k records in Oracle, if the number of rows processed in tMap equals to an amount of rows (for example: for each 1000 records), the tOracleInput sleeps and wait for the next iteration. 
Any suggestion will be appreciate. Thanks so much!  
Highlighted
Moderator

Re: ETL - Improve speed of process

Hi,
For your "OutofMemory" exception, could you please take a look at KB article:TalendHelpCenter:Exception: outOfMemory to see if it works?
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.

2019 GARTNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

Best Practices for Using Context Variables with Talend – Part 2

Part 2 of a series on Context Variables

Blog

Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables

Blog

Migrate Data from one Database to another with one Job using the Dynamic Schema

Find out how to migrate from one database to another using the Dynamic schema

Blog