I am having a job which will join 3 tables and load the data to the destination table, My source table is having 900000 records, when i execute the job it is showing an error as "Exception in thread "main" java.lang.Error: java.lang.Error: java.lang.OutOfMemoryError: GC overhead limit exceeded", How to solve this issue. Attached screen shot of the mapping.
I'm using TOS_DQ-5.4.2 on a MySQL database. How can I increase the heap size in this version as I don't see the interface shown in the KB article https://community.talend.com/t5/Migration-Configuration-and/OutOfMemory-Exception/ta-p/21669?content.... My ultimate goal is to find the number of duplicates in a large table (8 million rows, 128 fields). I'm starting to think this might not be possible with Talend even with a heap increase. But I also tried running a column set analysis on just 10 of the fields and I still got an out of heap space error. 6 of 10 fields were integer fields and the others were varchar. I also had an issue running column set analysis on a single column that was a big integer. Thanks for any help!
hi! you can solve the problem selecting only the fields that you need to process or selecting a limit number or rows. ex: select row_id from yourtable; or select * from yourtable where rownum<1000000; hope this help!
Thanks for the reply @eduardoAV. Unfortunately, I need to check for duplicates across all of the rows and across most of the fields. 3 fields are unnecessary but that still leaves me 125 fields, and I haven't found a way for Talend to process it.
hi, There are three way to increase Heap size: 1) in in TOOL Advance setting of run tab. 2) You can go to : window--> preference----> talend---> run/debug But it is not recommended, this will be applicable for all jobs you design. 3) export job and change value in shell script and run