I am loading around 7.5 million records from a db and after transformation the size of the records doubles to 15 million. But during execution my job got failed with Error "GC Overhead Limit exceeded on server". I am putting xmx - 10240 MB and xms - 1024 MB. Please find my job design below.
tmssqlinput-->tJavaRow-->tExtractJsonFields --> tMap --> tDenormalize --> tJavaRow --> tMSSQLOutput
At tExtractJSONFields data is getting double and at tDenormalize I am merging 2 records to One. I am getting this error at tDenormalize. Is there any better solution to this flow because tDenormalize is holding complete and then passing it one by one.
Could you please try with 4096 or bigger if You have this memory free on Talend Machine?
Introduction to Talend Open Studio for Data Integration.
Practical steps to developing your data integration strategy.
Create systems and workflow to manage clean data ingestion and data transformation.