tFileOutputExcel : issue with "big" files in format xlsx

[lennelei]

Hi all,
I try to make a quite simple job :
tOralceInput --> tFileOutputExcel
the tFileOutputExcel should write a xlsx file, and I've something like 100'000 or 200'000 rows only.
However, I couldn't find a way to make this work!
Without any specific settings, I've heap space errors :

 disconnected
Exception in thread "main" java.lang.Error: java.lang.OutOfMemoryError: Java heap space
	at ....tOracleInput_1Process(....java:1780)
	at ....runJobInTOS(....java:5109)
	at ....main(....java:4854)
Caused by: java.lang.OutOfMemoryError: Java heap space
	at java.util.Arrays.copyOf(Unknown Source)
	at java.io.ByteArrayOutputStream.write(Unknown Source)
	at org.apache.poi.openxml4j.opc.internal.MemoryPackagePartOutputStream.write(MemoryPackagePartOutputStream.java:88)
	at org.apache.xmlbeans.impl.store.Cursor._save(Cursor.java:590)
	at org.apache.xmlbeans.impl.store.Cursor.save(Cursor.java:2544)
	at org.apache.xmlbeans.impl.values.XmlObjectBase.save(XmlObjectBase.java:180)
	at org.apache.poi.xssf.usermodel.XSSFSheet.write(XSSFSheet.java:2480)
	at org.apache.poi.xssf.usermodel.XSSFSheet.commit(XSSFSheet.java:2439)
	at org.apache.poi.POIXMLDocumentPart.onSave(POIXMLDocumentPart.java:196)
	at org.apache.poi.POIXMLDocumentPart.onSave(POIXMLDocumentPart.java:200)
	at org.apache.poi.POIXMLDocument.write(POIXMLDocument.java:204)
	at org.talend.ExcelTool.writeExcel(ExcelTool.java:271)
	at ....tOracleInput_1Process(....java:1738)
	... 2 more

And when I try to use an output stream, job never ends...
I also try to use cursor on the tOracleInput but it crashes as well.
Any idea I could try to make this works?
Thanks
Regards
Alexis

[lennelei]

And you do extract on xlsx format and with Xmx1024 ?

[Anonymous]

Hi
Yes. XMX1024M.
Regards,
Pedro

[lennelei]

is it possible for you to send me the code of your job plz?

[Anonymous]

Hi
I have sent you an email and attach the code of my job.
Regards,
Pedro

[lennelei]

Hi, thanks for your code
Do you think the difference could be related to your "big data" version of TIS?

[Anonymous]

Hi
No. I have tried to create this job under several versions of TOS and TIS.
I think you'd better report it on BugTracker.
Regards,
Pedro

[lennelei]

Ok, last question: did you try with multiple columns? I managed to run the job with only one column as in your own job.

[Anonymous]

Hi
How many columns?
If this error only occurs when you add two many columns, it is because the job exceeds the JVM memory.
It's not a bug.
Regards,
Pedro

[lennelei]

5 columns
The point is that I don't understand why it takes up to 1GiB and 419s to fail to write to xlsx while it takes less than 2 seconds to write the same to a csv and less than 5 seconds and 87 MiB in memory for an xml ?

[Anonymous]

Hi
I don't understand, yet.
Sorry. I try to reproduce this issue but all failed.
Regards,
Pedro