What is amount of RAM is required to process 60 MB excel file

One Star

What is amount of RAM is required to process 60 MB excel file

Hi,
I am able to process 17 mB file by setting JVM setting to -Xmx2048M but When I am processing 52 MB excel file It's gives me the MemoryError - java heap space error. Then I trying various combination of JVM setting and maximum was -Xmx8192M but I am still getting same error 
I was initially using tFileInputExcel component later I also tried tFileExcelWorkbookOpen with tFileExcelsheetInput but getting same error
I am using Win7 64 bit with 8GB RAM 
Can any one tell me whether current RAM memory(8GB) is enough to process 52 MB excel file 
And what is the proportion of RAM Talend utilize to process per MB of excel file So I can have some ratio for future reference.
Below is the complete error
 connecting to socket on port 3863
connected
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at org.apache.xmlbeans.impl.store.Cur$CurLoadContext.attr(Cur.java:3044)
at org.apache.xmlbeans.impl.store.Cur$CurLoadContext.attr(Cur.java:3065)
at org.apache.xmlbeans.impl.store.Locale$SaxHandler.startElement(Locale.java:3263)
at org.apache.xmlbeans.impl.piccolo.xml.Piccolo.reportStartTag(Piccolo.java:1082)
at org.apache.xmlbeans.impl.piccolo.xml.PiccoloLexer.parseAttributesNS(PiccoloLexer.java:1802)
at org.apache.xmlbeans.impl.piccolo.xml.PiccoloLexer.parseOpenTagNS(PiccoloLexer.java:1521)
at org.apache.xmlbeans.impl.piccolo.xml.PiccoloLexer.parseTagNS(PiccoloLexer.java:1362)
at org.apache.xmlbeans.impl.piccolo.xml.PiccoloLexer.parseXMLNS(PiccoloLexer.java:1293)
at org.apache.xmlbeans.impl.piccolo.xml.PiccoloLexer.parseXML(PiccoloLexer.java:1261)
at org.apache.xmlbeans.impl.piccolo.xml.PiccoloLexer.yylex(PiccoloLexer.java:4812)
at org.apache.xmlbeans.impl.piccolo.xml.Piccolo.yylex(Piccolo.java:1290)
at org.apache.xmlbeans.impl.piccolo.xml.Piccolo.yyparse(Piccolo.java:1400)
disconnected
at org.apache.xmlbeans.impl.piccolo.xml.Piccolo.parse(Piccolo.java:714)
at org.apache.xmlbeans.impl.store.Locale$SaxLoader.load(Locale.java:3479)
at org.apache.xmlbeans.impl.store.Locale.parseToXmlObject(Locale.java:1277)
at org.apache.xmlbeans.impl.store.Locale.parseToXmlObject(Locale.java:1264)
at org.apache.xmlbeans.impl.schema.SchemaTypeLoaderBase.parse(SchemaTypeLoaderBase.java:345)
at org.openxmlformats.schemas.spreadsheetml.x2006.main.WorksheetDocument$Factory.parse(Unknown Source)
at org.apache.poi.xssf.usermodel.XSSFSheet.read(XSSFSheet.java:194)
at org.apache.poi.xssf.usermodel.XSSFSheet.onDocumentRead(XSSFSheet.java:186)
at org.apache.poi.xssf.usermodel.XSSFWorkbook.onDocumentRead(XSSFWorkbook.java:354)
at org.apache.poi.POIXMLDocument.load(POIXMLDocument.java:166)
at org.apache.poi.xssf.usermodel.XSSFWorkbook.<init>(XSSFWorkbook.java:240)
at org.apache.poi.xssf.usermodel.XSSFWorkbook.<init>(XSSFWorkbook.java:296)
at ifp.test_euro_alcohol_0_1.test_Euro_Alcohol.tFileInputExcel_1Process(test_Euro_Alcohol.java:746)
at ifp.test_euro_alcohol_0_1.test_Euro_Alcohol.runJobInTOS(test_Euro_Alcohol.java:1710)
at ifp.test_euro_alcohol_0_1.test_Euro_Alcohol.main(test_Euro_Alcohol.java:1567)


Moderator

Re: What is amount of RAM is required to process 60 MB excel file

Hi,
For your "Exception in thread "main" java.lang.OutOfMemoryError: Java heap space" issue, have you already checked the resolution in this KB article about:TalendHelpCenter:Exception Could not reserve enough space for object heap to see if it works?
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Six Stars JR
Six Stars

Re: What is amount of RAM is required to process 60 MB excel file

Just to make sure: you are using the 64bit version of TOS as well as a 64bit version of Java, correct?
One Star

Re: What is amount of RAM is required to process 60 MB excel file

Hi,
For your "Exception in thread "main" java.lang.OutOfMemoryError: Java heap space" issue, have you already checked the resolution in this KB article about:TalendHelpCenter:Exception+Could+not+reserve+enough+space+for+object+heap to see if it works?
Best regards
Sabrina

Hi I went through this article as it suggests I already have 64 bit jdk but still my Talend job throws error.
Initially I was getting this error while processing 20 MB file then I increased jvm parameter to -Xmx4096M then it worked 
but next time for 54 MB file it's failing then I tried incresing jvm parameter to  -Xmx5190M  but still job is failing with heap space error
I just wanted to know if my physical ram size is a challenge here to process the job ?
One Star

Re: What is amount of RAM is required to process 60 MB excel file

Just to make sure: you are using the 64bit version of TOS as well as a 64bit version of Java, correct?

yes, I am using the 64bit version of TOS as well as a 64bit version of Java
Six Stars JR
Six Stars

Re: What is amount of RAM is required to process 60 MB excel file

How many records do you have in your 60 MB Excel file? If it is an XLSX file then 60 MB is fairly big already and I cannot tell you if 8 GB of RAM will be enough. Based on your experience, it solely seems to be a RAM issue.
Moderator

Re: What is amount of RAM is required to process 60 MB excel file

Hi,
We see that you are using custom components tFileExcelWorkbookOpen and tFileExcelSheetInput from Talend Exchange?

Have you already checked out option (only for xlsx) to use the stream workbook implementation (memory saving mode) which does not hold all data ino the memory?
Or can you try to make conversion of the Excel files to CSV to see if it is OK with you?
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.