topic Re: How to read 1GB file in Talend Open Studio in Talend Studio

How to read 1GB file in Talend Open Studio

_AnonymousUser — Thu, 18 Apr 2013 16:18:02 GMT

Dear All,
While reading Excel File having 1GB data in Talend Open Studio I am getting Heep Space Error. Can any once help me this. How can I read big files in Talend Open Studio.
Thanks in advance!
Thanks & Regards
Chandra

Re: How to read 1GB file in Talend Open Studio

Anonymous — Thu, 18 Apr 2013 16:43:17 GMT

With the build-in Excel componente there is no way to do that, because they build one document first and then start iterating through it. For this use case please take a look at the componente suite tFileExcelWorkbook and tFileExcelSheetInput from Talend Exchange. The tFileExcelWorkbook component offers the option (only for xlsx) to use the stream workbook implementation (memory saving mode) which does not hold all data ino the memory.
For the OLE format xls there is unfortunately no way to read that without increasing the memory for the job as much as it needed.

Re: How to read 1GB file in Talend Open Studio

Anonymous — Fri, 19 Apr 2013 15:42:55 GMT

I think you need to be careful with the Excel components in Talend Open Studio. I have also been using large Excel files, but I ran into memory problems when using a very large Excel file with the .xlsx (XML) extension. Also using the binary version - with the .xlsb extension - gave problems, apparently because of the Apache POI library that is used in TOS. Conversion of the Excel files to CSV did solve these problems for me.

Re: How to read 1GB file in Talend Open Studio

Anonymous — Tue, 20 Aug 2013 22:37:03 GMT

hi All,
I am trying to load the data from one table in MSsql to another in redshift .But I am getting the error
Exception in component tRedshiftOutput_2
org.postgresql.util.PSQLException: ERROR: relation "demo" does not exist
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2096)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1829)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:510)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:386)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:332)
at demotalend.talenddem02_0_1.TalendDem02.tMSSqlInput_1Process(TalendDem02.java:2216)
at demotalend.talenddem02_0_1.TalendDem02.runJobInTOS(TalendDem02.java:2803)
at demotalend.talenddem02_0_1.TalendDem02.main(TalendDem02.java:2669)
can anyone one tell me why this error is coming.
Thanks
Saumya Sharma

Re: How to read 1GB file in Talend Open Studio

Anonymous — Wed, 21 Aug 2013 08:39:55 GMT

Hi Saumya Sharma,

ERROR: relation "demo" does not exist

The error imply relation "tablename" (table) don't exist for your output DB. Please pay attention on uppercase and lowercase letters in your DB.
Best regards
Sabrina