Out of Memory Error

One Star

Out of Memory Error

Hi,
We have developed an integration job to extract the daily updated data from Salesforce and sending it to our Datawarehousing team. The EDW team takes the extract file and load it into DB2 tables.
We received a new request to add few more fields to our existing job and we have completed the change. However, our EDW team wants the data for historical period and this is where we are facing the problem. In current production, the extract job runs for a period of 24 hours and now we are trying to run the job for 30 days period. The start and end dates are taken from a params file.
However, when we run for the historical extract, the job is failing with an "Out of Memory" error. The error messages are below.
==========================================================================
Exception in thread "Thread-2" java.lang.Error: java.lang.Error: java.lang.Error: java.lang.Error: java.lang.Error: java.lang.Error: java.lang.Error: java.lang.Error: java.lang.Error: java.lang.Error: java.lang.Error: java.lang.Error: java.lang.Error: java.lang.Error: java.lang.Error: java.lang.Error: java.lang.OutOfMemoryError: GC overhead limit exceeded
at omsleo.bna_batch_sf_edw_orderext_10152013_0_1.BNA_BATCH_SF_EDW_OrderExt_10152013.tJava_1Process(BNA_BATCH_SF_EDW_OrderExt_10152013.java:2684)
at omsleo.bna_batch_sf_edw_orderext_10152013_0_1.BNA_BATCH_SF_EDW_OrderExt_10152013$9.run(BNA_BATCH_SF_EDW_OrderExt_10152013.java:3049)
Caused by: java.lang.Error: java.lang.Error: java.lang.Error: java.lang.Error: java.lang.Error: java.lang.Error: java.lang.Error: java.lang.Error: java.lang.Error: java.lang.Error: java.lang.Error: java.lang.Error: java.lang.Error: java.lang.Error: java.lang.Error: java.lang.OutOfMemoryError: GC overhead limit exceeded
==========================================================================
I have to admit that our job is complicated, but it is failing irrespective of the tMap settings. I changed the tMap settings to "Write to disk" and provided the directory as /talend/leo/omsleo/tmp, but the job is still failing.
How can I get rid of this Out of Memory error? I changed the JVM arguments to 4096M also and that did not help.
The server that we deployed our job has 4GB RAM on it.
Can some one help?
Thanks,
Balaji.
One Star

Re: Out of Memory Error

FYI, I am using TIS 4.2.4 and we have no plans to upgrade until the first quarter of 2014 since we have more than 100 jobs running in production with TIS 4.2.4.
Thanks,
Balaji.
Five Stars

Re: Out of Memory Error

Have a read through this http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html on Java Garbage Collection.
Try increasing your HEAP.
Run->Advanced Settings.
Change -Xmx1024M to something bigger. Unless, of course, you've already been doing this...
You can also set this from the command line.
Moderator

Re: Out of Memory Error

Hi,
For your "Out of Memory", please have a look at the KB article on Talend Help Center ExceptionOutOfMemory, hope it will be helpful for you.
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: Out of Memory Error

Hi,
Thanks for the responses.
I have already set the heap size to 4096M in the script that launches the talend job.
I have requested the UNIX team to increase the memory on the box and they added a 16Gig swap space to the server. I did set the tMap paramter "store data to disk" already, but I set to a directory, rather than file. I don't know this will be an issue.
After doing all this, I re-ran the job and it failed. I am running the job for 15 days period now and it is failing multiple times with "Connection Reset" error message. I did see that the job uses several tSalesforceConnection objects to establish the connections and uses one connection for each subjob. All subjobs are connected using OnSubjobOk. I am assuming that when a subjob runs for a longer duration, the other tSalesforceConnection may be getting reset. I made the following changes.
1. Changed the job to use only one tSalesforceConnection object.
2. Checked the "Use Soap Compression" option on the tSalesforceConnection object.
I will try these changes sometime today and see how if it works.
Meanwhile, if any one has more thoughts, please feel free to chime in.
Thanks,
Balaji.
One Star

Re: Out of Memory Error

Some suggestions - might not work but worth trying..
1. If you are using tJavaRow and contacting strings with "+" will also consume huge amount of memory
Best Practices for Java String Concatenation:
When you have static strings (Ex: ?test?) it is ok to use "+" for concatenation. During the byte code conversion Java converts to String builder for the static strings.
When you have 4 or more dynamic strings it is best to use String builder. String objects are immutable. Using '+' will create a new instance of string object for every operand and will affect performance.
Example:
StringBuilder tmp = new StringBuilder(); // Using default 16 character size
tmp.append("first ");
tmp.append(variable1);
tmp.append(" middle ");
tmp.append(variable2);
tmp.append(" end");
return tmp.toString();
2. Try removing tLogRow's if not required
3. If you are running the jobs with the TOS client, then try running the jobs using batch or shell instead of standalone.
Seventeen Stars

Re: Out of Memory Error

1. That is not the case anymore in newer Java releases.
"s1" + "s2" + "s3"
becomes
new StringBuilder("s2").append("s2").append("s3").toString()
2. advise is a good advise. The problem is caused by to much so called transfer objects with a very short life time.
3. is not helpful because the studio is starting a separate VM for every new job start and that is nearly exactly the same like exporting the job and start it as standalone application.