tFileOutputExcel : issue with "big" files in format xlsx
Hi all,
I try to make a quite simple job :
tOralceInput --> tFileOutputExcel
the tFileOutputExcel should write a xlsx file, and I've something like 100'000 or 200'000 rows only.
However, I couldn't find a way to make this work!
Without any specific settings, I've heap space errors :
disconnected Exception in thread "main" java.lang.Error: java.lang.OutOfMemoryError: Java heap space at ....tOracleInput_1Process(....java:1780) at ....runJobInTOS(....java:5109) at ....main(....java:4854) Caused by: java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Unknown Source) at java.io.ByteArrayOutputStream.write(Unknown Source) at org.apache.poi.openxml4j.opc.internal.MemoryPackagePartOutputStream.write(MemoryPackagePartOutputStream.java:88) at org.apache.xmlbeans.impl.store.Cursor._save(Cursor.java:590) at org.apache.xmlbeans.impl.store.Cursor.save(Cursor.java:2544) at org.apache.xmlbeans.impl.values.XmlObjectBase.save(XmlObjectBase.java:180) at org.apache.poi.xssf.usermodel.XSSFSheet.write(XSSFSheet.java:2480) at org.apache.poi.xssf.usermodel.XSSFSheet.commit(XSSFSheet.java:2439) at org.apache.poi.POIXMLDocumentPart.onSave(POIXMLDocumentPart.java:196) at org.apache.poi.POIXMLDocumentPart.onSave(POIXMLDocumentPart.java:200) at org.apache.poi.POIXMLDocument.write(POIXMLDocument.java:204) at org.talend.ExcelTool.writeExcel(ExcelTool.java:271) at ....tOracleInput_1Process(....java:1738) ... 2 more
And when I try to use an output stream, job never ends...
I also try to use cursor on the tOracleInput but it crashes as well.
Any idea I could try to make this works?
Thanks
Regards
Alexis
Hi
Which version of Tos do you use?
Recently some users reported that tFileOutputExcel costed too much memory.
I doubt maybe it's a bug.
Besides, do you check the option "appen existing file"?
Regards,
Pedro
Hi
It's werid that tFileOutputExcel cost so much memory for recent version.
But I didn't reproduce this issues. They are all reported by users.
Please try to increase the JVM argument.
If it doesn't work, please report it on BugTracker.
Regards,
Pedro
Did you try a very simple test:
tRowGenerator (150 000 rows with 5 or 6 random Ascii strings) ---> tFileOutputExcel (xslx format)
On my computer (directly runned from studio), it crashes:
Démarrage du job test a 14:47 01/08/2012. connecting to socket on port 4064 connected Exception in thread "main" java.lang.Error: java.lang.OutOfMemoryError: Java heap space at transverse.test_0_1.test.tRowGenerator_1Process(test.java:804) at transverse.test_0_1.test.runJobInTOS(test.java:1067) at transverse.test_0_1.test.main(test.java:850) Caused by: java.lang.OutOfMemoryError: Java heap space at org.apache.xmlbeans.impl.store.Saver$TextSaver.resize(Saver.java:1592) disconnected at org.apache.xmlbeans.impl.store.Saver$TextSaver.preEmit(Saver.java:1223) at org.apache.xmlbeans.impl.store.Saver$TextSaver.emit(Saver.java:1144) at org.apache.xmlbeans.impl.store.Saver$TextSaver.emitElement(Saver.java:926) at org.apache.xmlbeans.impl.store.Saver.processElement(Saver.java:456) at org.apache.xmlbeans.impl.store.Saver.process(Saver.java:307) at org.apache.xmlbeans.impl.store.Saver$TextSaver.saveToString(Saver.java:1727) at org.apache.xmlbeans.impl.store.Cursor._xmlText(Cursor.java:546) at org.apache.xmlbeans.impl.store.Cursor.xmlText(Cursor.java:2436) at org.apache.xmlbeans.impl.values.XmlObjectBase.xmlText(XmlObjectBase.java:1455) at org.apache.xmlbeans.impl.values.XmlObjectBase.toString(XmlObjectBase.java:1440) at org.apache.poi.xssf.model.SharedStringsTable.addEntry(SharedStringsTable.java:167) at org.apache.poi.xssf.usermodel.XSSFCell.setCellValue(XSSFCell.java:345) at org.apache.poi.xssf.usermodel.XSSFCell.setCellValue(XSSFCell.java:315) at org.talend.ExcelTool.addCellValue(ExcelTool.java:250) at transverse.test_0_1.test.tRowGenerator_1Process(test.java:733) ... 2 more Job test terminé à 14:54 01/08/2012.
