Five Stars

How to handle many jobs for java GC Error(heap size)

Hi. I working on a TO S.

 

My Job project has lots of Jobs.(below image)

 

jobList.PNG

 

 

 

 

 

 

 

You should look at the left image in a "FRONTDOM_18" Folder, it has 36 Jobs(child job), 2 Starter(Parent job)

 

 

 

 

 

 

 

 

and each job is configured below image.

 

 

child.PNG

 [ setting : Multi thread execution(ON), db connection(shared), db commit(on), db close(off) in the child job) 

 

 

and starter ( parent job )

 

 

 

parentparent

 [ setting : Multi thread execution(ON), db connection(shared), db close(on) in the child job, Xms 2G, Xmx 4G) 

 

 

Q, There is no problem when running separately. however, errors occur only when they are bind and executed.

I think It's obviously a problem with memory, but i don't know how deal with this problem. 

Is there anything else besides increasing memory?

 

Q, If solution exists, can i run the Jobs Folder List(same time) on the first image

 

 

 

sorry for bad english skill;

thanks.

 

 

 

 

 

Exception in thread "Thread-122" java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.Arrays.copyOf(Unknown Source)
at java.lang.StringCoding.safeTrim(Unknown Source)
at java.lang.StringCoding.access$300(Unknown Source)
at java.lang.StringCoding$StringEncoder.encode(Unknown Source)
at java.lang.StringCoding.encode(Unknown Source)
at java.lang.String.getBytes(Unknown Source)
at org.postgresql.core.Utils.encodeUTF8(Utils.java:54)
at org.postgresql.core.v3.SimpleParameterList.getV3Length(SimpleParameterList.java:234)
at org.postgresql.core.v3.QueryExecutorImpl.sendBind(QueryExecutorImpl.java:808)
at org.postgresql.core.v3.QueryExecutorImpl.sendOneQuery(QueryExecutorImpl.java:1053)
at org.postgresql.core.v3.QueryExecutorImpl.sendQuery(QueryExecutorImpl.java:644)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:345)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeBatch(AbstractJdbc2Statement.java:2674)
at cdc.frontdom_193_198_0_1.FRONTDOM_193_198.tPostgresPlusInput_3Process(FRONTDOM_193_198.java:7841)
at cdc.frontdom_193_198_0_1.FRONTDOM_193_198$4.run(FRONTDOM_193_198.java:15021)
Batch entry 43 <unknown> was aborted. Call getNextException to see the cause.
18_130 : 43
Exception in component tPostgresPlusInput_6 (FRONTDOM_193_198)
org.postgresql.util.PSQLException: Ran out of memory retrieving query results.
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1309)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:192)
at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:451)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:336)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:235)
at cdc.frontdom_193_198_0_1.FRONTDOM_193_198.tPostgresPlusInput_6Process(FRONTDOM_193_198.java:14041)
at cdc.frontdom_193_198_0_1.FRONTDOM_193_198$7.run(FRONTDOM_193_198.java:15153)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
at org.postgresql.core.PGStream.ReceiveTupleV3(PGStream.java:349)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1306)
... 6 more
18_160 : 296
18_192 : 661
18_180 : 17
18_125 : 171

Exception in component tRunJob_273 (FRONTDOM_STARTER_2) java.lang.RuntimeException: Child job running failed. org.postgresql.util.PSQLException: Ran out of memory retrieving query results. at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1309) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:192) at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:451) at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:336) at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:235) at cdc.frontdom_193_198_0_1.FRONTDOM_193_198.tPostgresPlusInput_6Process(FRONTDOM_193_198.java:14041) at cdc.frontdom_193_198_0_1.FRONTDOM_193_198$7.run(FRONTDOM_193_198.java:15153) Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded at org.postgresql.core.PGStream.ReceiveTupleV3(PGStream.java:349) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1306) ... 6 more at cdc.frontdom_starter_2_0_1.FRONTDOM_STARTER_2.tRunJob_273Process(FRONTDOM_STARTER_2.java:4792) at cdc.frontdom_starter_2_0_1.FRONTDOM_STARTER_2$15.run(FRONTDOM_STARTER_2.java:6436) 18_186 : 532041 FRONTDOM_Stater is completed 521088 milliseconds  

 

1 ACCEPTED SOLUTION

Accepted Solutions
Five Stars

Re: How to handle many jobs for java GC Error(heap size)

Thank to reply,

 

but, I search the error message from my question,

 

I found it UseGCOverheadLimit option is just error ignore(masking)..

 

I think preparedstatemts cache size. 

 

Anyway, I solved the problem with Batch Size = 10000 to 5000,  exec time little bit slow, but error does not appears,

 

Thank to reply

2 REPLIES
Six Stars

Re: How to handle many jobs for java GC Error(heap size)

Hi

please try the Following workaround to solved the problem in Talend 

Add new option to Windows–>Preferences–>Talend–>Run/Debug  – XX:-UseGCOverheadLimit

Five Stars

Re: How to handle many jobs for java GC Error(heap size)

Thank to reply,

 

but, I search the error message from my question,

 

I found it UseGCOverheadLimit option is just error ignore(masking)..

 

I think preparedstatemts cache size. 

 

Anyway, I solved the problem with Batch Size = 10000 to 5000,  exec time little bit slow, but error does not appears,

 

Thank to reply