java.lang.OutOfMemoryError and tFileInputExcel

[Rosanero4Ever]

Hi all,
I'm using a tFileInputExcel in order to import an Excel file (about 25k rows and 10 columns) in my DB.
I'm using TOS 5.1.2 on Windows 2008 R2 in a computer provided with 8GB RAM
My memory configuration is the following:
-vmargs
-Xms64m
-Xmx1536m
-XX:MaxPermSize=512m
-Dfile.encoding=UTF-8
Despite of this I have the following exception:
Exception in thread "main" java.lang.Error: java.lang.Error: java.lang.Error: java.lang.Error: java.lang.Error: java.lang.Error: java.lang.OutOfMemoryError: Java heap space at
I use store data functionality of tMap and commit my data every 1000 rows but I yet have this problem.
I read some posts about this issue and I applied all the advices.
I also edited the memory configuration with Xmx=3096m but...nothing 😞
I wouldn't convert my excel in csv.
So, can anybody help me to find a solution, please?

[Anonymous]

The built-in Excel components from Talend use a very - let me say - poor library for the xls format. It would be much better if they moved to the Apache POI library as they do already for the xlsx format.
You can try the combination of tFileExcelWorkbookOpen and tFileExcelSheetInput (both working together and use the latest Apache POI lib V3.8 and available over Talend Exchange) to read out your sheet. This has also the advantage, you can exactly specify which column you want to read.
If have read such huge files with the help of this components.

[Anonymous]

hi all,
I think that Talend 5.1.2 use POI lib to read Excel file .

regards
laurent

[Anonymous]

Yes but it looks like only for the new xlsx format (the only good news is that we have v3.8 now). For xls they use the classes from the jxl lib.

[Rosanero4Ever]

Hi all,
my Excel file has xlsx extension, so new library shoud be used. Isn't it?

[Anonymous]

Yes, it is. It would be interesting to know how large your file was. It have tested all components with a very large file of nearly 15MB file size with 20 sheets and in sum more than 65535 rows.

[Rosanero4Ever]

my file is about 400k rows in a single sheet (100MB)
Thanks a lot

[Anonymous]

You should close this thread as resolved.

[Rosanero4Ever]

Sorry..
I can close this thread as resolved,
but I don't understand how the issue I wrote was resolved
...It was resolved?

[Anonymous]

No it isn't. But to be onest an Excel file with 100MB is a bit .... I think there will be hardly any solution for it.
Your solution to use csv format instead of the excel format is the best way from my point of view.

[Rosanero4Ever]

You're right 🙂
So, in your opinion, what order of size would I consider to use this component without using csv?
Thank a lot for your replies.