One Star

Out of Heap Space

Hi,

I am having a job which will join 3 tables and load the data to the destination table,
My source table is having 900000 records, when i execute the job it is showing an error as
"Exception in thread "main" java.lang.Error: java.lang.Error: java.lang.OutOfMemoryError: GC overhead limit exceeded", How to solve this issue. Attached screen shot of the mapping.
5 REPLIES
Community Manager

Re: Out of Heap Space

Hi
Take a look at this KB article to find a solution to reslove this issue.
Shong
----------------------------------------------------------
Talend | Data Agility for Modern Business
One Star

Re: Out of Heap Space

I'm using TOS_DQ-5.4.2 on a MySQL database. How can I increase the heap size in this version as I don't see the interface shown in the KB article https://community.talend.com/t5/Migration-Configuration-and/OutOfMemory-Exception/ta-p/21669?content....
My ultimate goal is to find the number of duplicates in a large table (8 million rows, 128 fields).
I'm starting to think this might not be possible with Talend even with a heap increase. But I also tried running a column set analysis on just 10 of the fields and I still got an out of heap space error. 6 of 10 fields were integer fields and the others were varchar.
I also had an issue running column set analysis on a single column that was a big integer.
Thanks for any help!
Four Stars

Re: Out of Heap Space

hi! you can solve the problem selecting only the fields that you need to process or selecting a limit number or rows. ex:
select row_id from yourtable;
or
select * from yourtable where rownum<1000000;
hope this help!
One Star

Re: Out of Heap Space

Thanks for the reply @eduardoAV. Unfortunately, I need to check for duplicates across all of the rows and across most of the fields. 3 fields are unnecessary but that still leaves me 125 fields, and I haven't found a way for Talend to process it.
One Star

Re: Out of Heap Space

hi,
There are three way to increase Heap size:
1) in in TOOL Advance setting of run tab.
2) You can go to : window--> preference----> talend---> run/debug
But it is not recommended, this will be applicable for all jobs you design.
3) export job and change value in shell script and run

Regards,
Akshath