How to read 1GB file in Talend Open Studio

One Star

How to read 1GB file in Talend Open Studio

Dear All,
While reading Excel File having 1GB data in Talend Open Studio I am getting Heep Space Error. Can any once help me this. How can I read big files in Talend Open Studio.
Thanks in advance!
Thanks & Regards
Chandra
Seventeen Stars

Re: How to read 1GB file in Talend Open Studio

With the build-in Excel componente there is no way to do that, because they build one document first and then start iterating through it. For this use case please take a look at the componente suite tFileExcelWorkbook and tFileExcelSheetInput from Talend Exchange. The tFileExcelWorkbook component offers the option (only for xlsx) to use the stream workbook implementation (memory saving mode) which does not hold all data ino the memory.
For the OLE format xls there is unfortunately no way to read that without increasing the memory for the job as much as it needed.
One Star

Re: How to read 1GB file in Talend Open Studio

I think you need to be careful with the Excel components in Talend Open Studio. I have also been using large Excel files, but I ran into memory problems when using a very large Excel file with the .xlsx (XML) extension. Also using the binary version - with the .xlsb extension - gave problems, apparently because of the Apache POI library that is used in TOS. Conversion of the Excel files to CSV did solve these problems for me.
One Star

Re: How to read 1GB file in Talend Open Studio

hi All,
I am trying to load the data from one table in MSsql to another in redshift .But I am getting the error
Exception in component tRedshiftOutput_2
org.postgresql.util.PSQLException: ERROR: relation "demo" does not exist
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2096)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1829)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:510)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:386)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:332)
at demotalend.talenddem02_0_1.TalendDem02.tMSSqlInput_1Process(TalendDem02.java:2216)
at demotalend.talenddem02_0_1.TalendDem02.runJobInTOS(TalendDem02.java:2803)
at demotalend.talenddem02_0_1.TalendDem02.main(TalendDem02.java:2669)
can anyone one tell me why this error is coming.
Thanks
Saumya Sharma
Moderator

Re: How to read 1GB file in Talend Open Studio

Hi Saumya Sharma,
ERROR: relation "demo" does not exist

The error imply relation "tablename" (table) don't exist for your output DB. Please pay attention on uppercase and lowercase letters in your DB.
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.