GC overhead limit error when running Jobs in Studio

Symptoms

You may have a Job that uses a lot of resources as it attempts to run. When the Job is run in Studio or against a remote JobServer, it errors out. Upon viewing the execution logs, you may see the following error:

Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded 
at java.util.Arrays.copyOf(Unknown Source) 
at java.util.ArrayList.grow(Unknown Source) 
at java.util.ArrayList.ensureExplicitCapacity(Unknown Source) 
at java.util.ArrayList.ensureCapacityInternal(Unknown Source) 
at java.util.ArrayList.add(Unknown Source) 
at org.apache.xmlbeans.impl.store.Saver.addMapping(Saver.java:641) 
at org.apache.xmlbeans.impl.store.Saver.<init>(Saver.java:90) 
at org.apache.xmlbeans.impl.store.Saver$TextSaver.<init>(Saver.java:916) 
at org.apache.xmlbeans.impl.store.Cursor._xmlText(Cursor.java:546) 
at org.apache.xmlbeans.impl.store.Cursor.xmlText(Cursor.java:2436) 
at org.apache.xmlbeans.impl.values.XmlObjectBase.xmlText(XmlObjectBase.java:1500) 
at org.apache.poi.xssf.model.SharedStringsTable.getKey(SharedStringsTable.java:134) 
at org.apache.poi.xssf.model.SharedStringsTable.readFrom(SharedStringsTable.java:124) 
at org.apache.poi.xssf.model.SharedStringsTable.<init>(SharedStringsTable.java:106) 
... 14 more 

 

Diagnosis

The issue is that the Job is attempting to use more memory that it can access by default. This may happen regardless of how much memory has been allocated for the JobServer, for example 32GB on the JobServer alone. This error can be caused when the Job is using 98% or more of the default allocated heap space that is allowed for the JobServer, usually defined when the JobServer, TAC, or other Talend module is started using the -XX:MaxHeapSpace flag.

 

Solution

If a Job uses a large amount of memory, to resolve the issue and remove the error, add the following flag:

-XX:-UseGCOverheadLimit

When the Job runs, this flag will bypass the JVM cache limit and use as much memory as the Job needs to run.

 

How to add a flag to the Job

There are two ways the flag can be added so that it will be activated when the Job runs.

 

Add the flag to the JVM parameters attached to the task in Job Conductor.

  1. In Job Conductor, select the affected task that the flag is to be used on.
  2. Click the JVM parameters on the bottom. The -XX:-UseGCOverheadLimit flag can now be added to the task; the next time the Job runs, the flag will be bundled with the task, and it will bypass the memory error.

    Note: This option only applies to the task; if the task is removed from Job Conductor, this flag will be removed. This can be useful for those instances when the task gets this error due to the Job being large only temporarily.

 

Add the flag to the Job

If the Job regularly requires a lot of resources, or if you don't want to have to keep adding the flag to each new task in Job Conductor, add the flag to the Job itself.

  1. Open the Job in Studio, and navigate to the Advanced > Run tab below the designer view.
  2. Click the plus [+] sign next to the JVM parameters window, and a dialog will open where you can add the -XX:-UseGCOverheadLimit flag.

    Add JVM Parameters to a job via Studio.PNG

     

  3. Click OK, then save the Job.

    The next time the Job is built/run, either in Studio or Job Conductor, the JVM parameter will be included in the Job itself so the flag will apply.

     

    Add JVM Parameters to a job via Studio 2.PNG

 

For more information on how to add JVM parameters to your tasks or Jobs, please see the following pages on the Talend Help Center:

Setting JVM parameters for specific tasks

How to set advanced execution settings

Version history
Revision #:
7 of 7
Last update:
‎09-28-2018 11:57 PM
Updated by: