tFileOutputExcel memory error

[AlessandroCic]

Hi forum

I have a problem using in a simply job how I write the result of a query in an Excel file.

If in the component I not set "Add to existing file" it works.

If I set "Add to existing file" I have this error:

Exception in thread "main" java.lang.: Java heap space

I tried to increase the memory value Xmx in the advanced setting of Run but without success.

So the data you want to write the files are so many, about 300k row and the dimension of the file Excel is 26Mb.

I work in this environment:

Lubuntu virtual machine 64bit

4Gb of memory

java version "1.8.0_102"
Java HotSpot(TM) 64-Bit Server VM (build 25.102-b14, mixed mode)

It it possible to resolve  this problem?

Regards

Alessandro

[AlessandroCic]

Hi

I'm very sure the file haven't more than one row.

1. Ctrl+Alt+End command stops on the last cell of the first line
2. "Full" file is 30MiB large, "empty" file only 6KiB

Yet I still have the error:

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
	at org.apache.xmlbeans.impl.store.Saver$TextSaver.resize(Saver.java:1700)
	at org.apache.xmlbeans.impl.store.Saver$TextSaver.preEmit(Saver.java:1303)
	at org.apache.xmlbeans.impl.store.Saver$TextSaver.emit(Saver.java:1190)
	at org.apache.xmlbeans.impl.store.Saver$TextSaver.emitElement(Saver.java:962)
	at org.apache.xmlbeans.impl.store.Saver.processElement(Saver.java:476)
	at org.apache.xmlbeans.impl.store.Saver.process(Saver.java:307)
	at org.apache.xmlbeans.impl.store.Saver$TextSaver.saveToString(Saver.java:1864)
	at org.apache.xmlbeans.impl.store.Cursor._xmlText(Cursor.java:546)
	at org.apache.xmlbeans.impl.store.Cursor.xmlText(Cursor.java:2436)
	at org.apache.xmlbeans.impl.values.XmlObjectBase.xmlText(XmlObjectBase.java:1500)
	at org.apache.poi.xssf.model.SharedStringsTable.getKey(SharedStringsTable.java:134)
	at org.apache.poi.xssf.model.SharedStringsTable.addEntry(SharedStringsTable.java:180)
	at org.apache.poi.xssf.usermodel.XSSFCell.setCellValue(XSSFCell.java:362)

In the last test I have increased the swap memory to disk to have the same value as physical memory (4Gb+4Gb)... unfortunately still unsuccessful.

I'm pretty sure the problem is in the component.

If I do not select "Add to existing file" in the advanced option I can set the number of row to the buffer.

I haven't this possibility if I select "Add to existing file"; when it runs store in the memory all the data until the end of the record-set but this way it uses all the memory available as long as it ends.

Instead with not select "Add to existing file", it writes data every XXX rows and releases memory for new data; I can see this because the size of the file it increase on the disk during processing.
I not think there is a solution unless you modify the component or find a way to configure the memory usage.

Regards

Alessandro

[TRF]

Well, I would like to be able to modify the component by myself, but I'm not from Talend...

Any chance to test with a PC with 8Gb RAM?