ETL - Improve speed of process

One Star

ETL - Improve speed of process

Dear friends, I have a problem in Talend Open Studio. My job is described as below: 
tOracleInput --->(1) TMap (Modify data) -->(2) tWriteJSONFields -->(3) tMongoDBOutput
In Oracle database, there are around 300k records.  The processing speed of (1) and (2) is around  3054,98rows/s, but in (3) is only around 10rows/s due to the process of tWriteJSONFields. The questions are: 
- How can I improve the speed of (3)? 
- Some time, there is an OutMemory Exception because data which is cached in (2) waiting for the process of tWriteJSONFields is over load. So can I limit a number of rows each time. More clearly, there are 300k records in Oracle, if the number of rows processed in tMap equals to an amount of rows (for example: for each 1000 records), the tOracleInput sleeps and wait for the next iteration. 
Any suggestion will be appreciate. Thanks so much!  
Moderator

Re: ETL - Improve speed of process

Hi,
For your "OutofMemory" exception, could you please take a look at KB article:TalendHelpCenter:Exception: outOfMemory to see if it works?
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.