OutofMemory Error When considering a huge XML File as source

One Star

OutofMemory Error When considering a huge XML File as source

Hi Shong,
I have a SDMX XML File as my source and earlier xml file size was in kb and able to run the jobs successfully.
Now we got a very huge 1GB XMl file and if run the same job job getting failed and throwing the below error.
Can you please suggest me with proper workarounds and their steps.
Please help me as it is very urgent and and life issue.
Error 1:
===========
Starting job j500_ValidateLoad at 14:51 12/07/2012.
connecting to socket on port 3958
connected
Exception in thread "Thread-7" java.lang.OutOfMemoryError: Java heap space
at java.util.LinkedList.listIterator(LinkedList.java:667)
at java.util.AbstractList.listIterator(AbstractList.java:284)
at java.util.AbstractSequentialList.iterator(AbstractSequentialList.java:222)
at routines.system.RunStat.sendMessages(RunStat.java:248)
at routines.system.RunStat.run(RunStat.java:212)
at java.lang.Thread.run(Thread.java:619)
Exception in thread "main" java.lang.Error: java.lang.Error: java.lang.Error: java.lang.Error: java.lang.Error: java.lang.OutOfMemoryError: Java heap space
at datasetimportjobs.j500_validateload_6_0.j500_ValidateLoad.tOracleInput_1Process(j500_ValidateLoad.java:4042)
at datasetimportjobs.j500_validateload_6_0.j500_ValidateLoad.runJobInTOS(j500_ValidateLoad.java:5436)
at datasetimportjobs.j500_validateload_6_0.j500_ValidateLoad.main(j500_ValidateLoad.java:5113)
Caused by: java.lang.Error: java.lang.Error: java.lang.Error: java.lang.Error: java.lang.OutOfMemoryError: Java heap space
disconnected
at datasetimportjobs.j100_read_files_6_1.j100_Read_Files.tRunJob_1Process(j100_Read_Files.java:2555)
at datasetimportjobs.j100_read_files_6_1.j100_Read_Files.runJobInTOS(j100_Read_Files.java:2942)
at datasetimportjobs.j100_read_files_6_1.j100_Read_Files.runJob(j100_Read_Files.java:2608)
at datasetimportjobs.j500_validateload_6_0.j500_ValidateLoad.tOracleInput_1Process(j500_ValidateLoad.java:3375)
... 2 more
Caused by: java.lang.Error: java.lang.Error: java.lang.Error: java.lang.OutOfMemoryError: Java heap space
at datasetimportjobs.j100_read_files_6_1.j100_Read_Files.tRunJob_2Process(j100_Read_Files.java:956)
at datasetimportjobs.j100_read_files_6_1.j100_Read_Files.tRunJob_1Process(j100_Read_Files.java:2546)
... 5 more
Caused by: java.lang.Error: java.lang.Error: java.lang.OutOfMemoryError: Java heap space
at datasetimportjobs.j102_read_dataset_xml_0_6.j102_Read_Dataset_XML.tFileList_1Process(j102_Read_Dataset_XML.java:1030)
at datasetimportjobs.j102_read_dataset_xml_0_6.j102_Read_Dataset_XML.runJobInTOS(j102_Read_Dataset_XML.java:1498)
at datasetimportjobs.j102_read_dataset_xml_0_6.j102_Read_Dataset_XML.runJob(j102_Read_Dataset_XML.java:1282)
at datasetimportjobs.j100_read_files_6_1.j100_Read_Files.tRunJob_2Process(j100_Read_Files.java:895)
... 6 more
Caused by: java.lang.Error: java.lang.OutOfMemoryError: Java heap space
at datasetimportjobs.j140_import_segments_0_6.j140_Import_Segments.tFileInputXML_1Process(j140_Import_Segments.java:2345)
at datasetimportjobs.j140_import_segments_0_6.j140_Import_Segments.runJobInTOS(j140_Import_Segments.java:2767)
at datasetimportjobs.j140_import_segments_0_6.j140_Import_Segments.runJob(j140_Import_Segments.java:2572)
at datasetimportjobs.j102_read_dataset_xml_0_6.j102_Read_Dataset_XML.tFileList_1Process(j102_Read_Dataset_XML.java:762)
... 9 more
Caused by: java.lang.OutOfMemoryError: Java heap space
at org.dom4j.DocumentFactory.createAttribute(DocumentFactory.java:156)
at org.dom4j.tree.AbstractElement.setAttributes(AbstractElement.java:549)
at org.dom4j.io.SAXContentHandler.addAttributes(SAXContentHandler.java:916)
at org.dom4j.io.SAXContentHandler.startElement(SAXContentHandler.java:249)
at org.apache.xerces.parsers.AbstractSAXParser.startElement(Unknown Source)
at org.apache.xerces.parsers.AbstractXMLDocumentParser.emptyElement(Unknown Source)
at org.apache.xerces.impl.XMLNSDocumentScannerImpl.scanStartElement(Unknown Source)
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source)
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)
at org.dom4j.io.SAXReader.read(SAXReader.java:465)
at datasetimportjobs.j140_import_segments_0_6.j140_Import_Segments.tFileInputXML_1Process(j140_Import_Segments.java:1601)
... 12 more
Job j500_ValidateLoad ended at 14:58 12/07/2012.
Error 2:
==========
Starting job j500_ValidateLoad at 15:23 12/07/2012.
connecting to socket on port 3527
connected
Exception in thread "main" java.lang.Error: java.lang.Error: java.lang.Error: java.lang.Error: java.lang.Error: java.lang.OutOfMemoryError: Java heap space
at datasetimportjobs.j500_validateload_6_0.j500_ValidateLoad.tOracleInput_1Process(j500_ValidateLoad.java:4042)
at datasetimportjobs.j500_validateload_6_0.j500_ValidateLoad.runJobInTOS(j500_ValidateLoad.java:5436)
at datasetimportjobs.j500_validateload_6_0.j500_ValidateLoad.main(j500_ValidateLoad.java:5113)
Caused by: java.lang.Error: java.lang.Error: java.lang.Error: java.lang.Error: java.lang.OutOfMemoryError: Java heap space
disconnected
at datasetimportjobs.j100_read_files_6_1.j100_Read_Files.tRunJob_1Process(j100_Read_Files.java:2555)
at datasetimportjobs.j100_read_files_6_1.j100_Read_Files.runJobInTOS(j100_Read_Files.java:2942)
at datasetimportjobs.j100_read_files_6_1.j100_Read_Files.runJob(j100_Read_Files.java:2608)
at datasetimportjobs.j500_validateload_6_0.j500_ValidateLoad.tOracleInput_1Process(j500_ValidateLoad.java:3375)
... 2 more
Caused by: java.lang.Error: java.lang.Error: java.lang.Error: java.lang.OutOfMemoryError: Java heap space
at datasetimportjobs.j100_read_files_6_1.j100_Read_Files.tRunJob_2Process(j100_Read_Files.java:956)
at datasetimportjobs.j100_read_files_6_1.j100_Read_Files.tRunJob_1Process(j100_Read_Files.java:2546)
... 5 more
Caused by: java.lang.Error: java.lang.Error: java.lang.OutOfMemoryError: Java heap space
at datasetimportjobs.j102_read_dataset_xml_0_6.j102_Read_Dataset_XML.tFileList_1Process(j102_Read_Dataset_XML.java:1030)
at datasetimportjobs.j102_read_dataset_xml_0_6.j102_Read_Dataset_XML.runJobInTOS(j102_Read_Dataset_XML.java:1498)
at datasetimportjobs.j102_read_dataset_xml_0_6.j102_Read_Dataset_XML.runJob(j102_Read_Dataset_XML.java:1282)
at datasetimportjobs.j100_read_files_6_1.j100_Read_Files.tRunJob_2Process(j100_Read_Files.java:895)
... 6 more
Caused by: java.lang.Error: java.lang.OutOfMemoryError: Java heap space
at datasetimportjobs.j140_import_segments_0_6.j140_Import_Segments.tFileInputXML_1Process(j140_Import_Segments.java:2345)
at datasetimportjobs.j140_import_segments_0_6.j140_Import_Segments.runJobInTOS(j140_Import_Segments.java:2767)
at datasetimportjobs.j140_import_segments_0_6.j140_Import_Segments.runJob(j140_Import_Segments.java:2572)
at datasetimportjobs.j102_read_dataset_xml_0_6.j102_Read_Dataset_XML.tFileList_1Process(j102_Read_Dataset_XML.java:762)
... 9 more
Caused by: java.lang.OutOfMemoryError: Java heap space
at org.dom4j.DocumentFactory.createAttribute(DocumentFactory.java:156)
at org.dom4j.tree.AbstractElement.setAttributes(AbstractElement.java:549)
at org.dom4j.io.SAXContentHandler.addAttributes(SAXContentHandler.java:916)
at org.dom4j.io.SAXContentHandler.startElement(SAXContentHandler.java:249)
at org.apache.xerces.parsers.AbstractSAXParser.startElement(Unknown Source)
at org.apache.xerces.parsers.AbstractXMLDocumentParser.emptyElement(Unknown Source)
at org.apache.xerces.impl.XMLNSDocumentScannerImpl.scanStartElement(Unknown Source)
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source)
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)
at org.dom4j.io.SAXReader.read(SAXReader.java:465)
at datasetimportjobs.j140_import_segments_0_6.j140_Import_Segments.tFileInputXML_1Process(j140_Import_Segments.java:1601)
... 12 more
Exception in thread "Thread-7" java.lang.OutOfMemoryError: Java heap space
at java.util.LinkedList.listIterator(LinkedList.java:667)
at java.util.AbstractList.listIterator(AbstractList.java:284)
at java.util.AbstractSequentialList.iterator(AbstractSequentialList.java:222)
at routines.system.RunStat.sendMessages(RunStat.java:248)
at routines.system.RunStat.run(RunStat.java:212)
at java.lang.Thread.run(Thread.java:619)
Job j500_ValidateLoad ended at 15:25 12/07/2012.
Thanks,
KPK
One Star

Re: OutofMemory Error When considering a huge XML File as source

Hi KPK
This is a common problem.
What's the volume of your memory?
What you can do now is trying to reduce the consumption of memory and increase the arguments of JVM.
If there is tMap component in your job, you need to use "Store on disk" feature.
You can increase the arguments of JVM as the following image.
Regards,
Pedro
One Star

Re: OutofMemory Error When considering a huge XML File as source

I have 10 Jobs calling in 1 Parent Job and in all the 10 jobs i have tmaps...Do i need to use Store on Disk for all the 10 Jobs or is it okay if i use in Parent Job.
Seventeen Stars

Re: OutofMemory Error When considering a huge XML File as source

hi all,

semms that you're using dom4j
try to use SAX parser ( advanced setting) .. it could solve problem for huge xml file. ...but it's slower Smiley Happy
regards
laurent
One Star

Re: OutofMemory Error When considering a huge XML File as source

Hi
I agree with Laurent.
All these two features are used to reduce the consuming of JVM memory.
Whether you use "Store on disk" for all child jobs depends on how you balance the performance and memory comsuming.
Regards,
Pedro
One Star

Re: OutofMemory Error When considering a huge XML File as source

Where can i find SAX Parser..Can u write me detailed steps to do that...
Seventeen Stars

Re: OutofMemory Error When considering a huge XML File as source

under advanced setting of your xml input component.
it's just an option in a list (menu) to choice
One Star

Re: OutofMemory Error When considering a huge XML File as source

I have made the suggested changes and run the Job....Still No Luck........
Can Some one Please help me....
Can we Split Large XML into Small Chunks (Multiple XML Files).......How can we do that...?
What will be the impact if we do like that
Seventeen Stars

Re: OutofMemory Error When considering a huge XML File as source

like Pedro've said try to change param jvm into .bat or .sh depending on our OS :
java -Xms256M -Xmx1024M ..... etc

can read some data in 5Gb xml file with that config ...
hope it's help
laurent
One Star

Re: OutofMemory Error When considering a huge XML File as source

Can You Please help me with the exact code...
One Star

Re: OutofMemory Error When considering a huge XML File as source

This didn't Work for me...............Still iam getting the same Error
One Star

Re: OutofMemory Error When considering a huge XML File as source

Any Help?????????
One Star

Re: OutofMemory Error When considering a huge XML File as source

I have 300 MB XML File and as suggested changed all the JVM arguments to 256/512/1024 and 1024,using SAX Parser and Still iam facing the same issue..can anyone please help me out with some workarounds!!
One Star

Re: OutofMemory Error When considering a huge XML File as source

Hi
So there is the component tRunjob in your parent job?
You might use the feature of tRunJob--Use an independent process to run subjob.
It will reduce the memory consuming. But make sure there is no context or data transmition between parent job and this subjob. Or you will get error.
Regards,
Pedro
One Star

Re: OutofMemory Error When considering a huge XML File as source

Pedro,
Context is there and data transmission happends.....
What will be the reason behind this......it is not able to handle 300MB file...then how it will handle large files of size 5 or 10GB???

Thanks,
KPK
One Star

Re: OutofMemory Error When considering a huge XML File as source

Hi KPK
All the parent job and the child jobs share only one JVM thread.
The maximum memory is the JVM argument xmx.
Try to increase the JVM arguments. If it doesn't work either you have to simplify your jobs.
The feature "Use an independent process to run subjob" will create a new JVM thread for the subjob. But you say you can't because you need to transmit context variables.
Regards,
Pedro