[resolved] java.lang.OutOfMemoryError: Java heap space

[Anonymous]

Hi All.
One Table have 4million rows.
I am Exporting 4million rows to csv file i am getting below error.
"Exception in thread "main" java.lang.OutOfMemoryError: Java heap space"
How to increase Java heap space?
Thanks
kumar.talend

[Anonymous]

Hi  VictorTsui01,
Does this " java.lang.OutOfMemoryError: Java heap space"" issue repro on all your jobs or a specific job?
Best regards
Sabrina

[tale103108]

Shong,  your link is broken....

---

@shongwrote:
Hi

I am modify the "TOS_DI-win32-x86.ini" configuration file based on your suggested article.

The "TOS_DI-win32-x86.ini" configuration file is for Talend Studio application itself, not for the job execution, don't know which article or blob have you read, please refer to the KB article Exception+outOfMemory to learn how to allocate more memory to the job execution.
Shong

---

[KarthikGs]

delete the cache file too

[Anonymous]

Hello,

Please check the KB article about:https://community.talend.com/t5/Migration-Configuration-and/OutOfMemory-Exception/ta-p/21669

Best regards

Sabrina

[Anonymous]

---

@xdshi wrote:

Hello,

Please check the KB article about:https://community.talend.com/t5/Migration-Configuration-and/OutOfMemory-Exception/ta-p/21669

Best regards

Sabrina

---

 Hi Sabrina,

I'm having the same error "Exception in thread "Thread-0" Exception in thread "main" java.lang.OutOfMemoryError: Java heap space". 

My OS is Windows 7 Professional Service Pack 1, 64 bits and my RAM 8GB.

I'm using Talend Open Studio for Data Integration, Version: 7.0.1

I'm trying to read an Excel file with 1 million rows and 34 columns aprox., using tFileInputExcel and a tLogRow, but my job only reads the first row (header) and then I get the error.

If the job succeed, I'll process the information with components such as tMap, tAggregateRow, tPivotToColumnsDelimited, tFilterRow, tHashInput and tHashOutput, sending the entire results to a tFileOutputExcel & tFileOutputDelimited.

My advanced settings are:

-Xsm256M

-Xmx1024M

 Any suggestion?

Thanks!

[Anonymous]

Hello @MayTorres

Have you already tried to allocate more memory for the Job to be able to process large amounts of data? Please try to remove the parameter -Xmx1024m and add a new one like -Xmx2048m.

Hope this article helps:https://stackoverflow.com/questions/14763079/what-are-the-xms-and-xmx-parameters-when-starting-jvms

Best regards

Sabrina

[Anonymous]

---

@xdshiwrote:

Hello @MayTorres

Have you already tried to allocate more memory for the Job to be able to process large amounts of data? Please try to remove the parameter -Xmx1024m and add a new one like -Xmx2048m.

Hope this article helps:https://stackoverflow.com/questions/14763079/what-are-the-xms-and-xmx-parameters-when-starting-jvms

Best regards

Sabrina

---

Hi @xdshi

Yes, I tried, but I got the error that I leave later. I also read the article and I do not see anything that can help me, I do not know if I'm missing something. I know that it is possible to process that amount of information and I can’t think what I should do. I'm new to Talend and Java ... any other ideas? Please.

Thanks!

"Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
[statistics] disconnected
    at com.sun.org.apache.xerces.internal.xni.XMLString.toString(Unknown Source)
    at com.sun.org.apache.xerces.internal.util.XMLAttributesImpl.getValue(Unknown Source)
    at com.sun.org.apache.xerces.internal.parsers.AbstractDOMParser.startElement(Unknown Source)
    at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.scanStartElement(Unknown Source)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(Unknown Source)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(Unknown Source)
    at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(Unknown Source)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
    at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source)
    at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source)
    at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(Unknown Source)
    at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(Unknown Source)
    at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(Unknown Source)
    at javax.xml.parsers.DocumentBuilder.parse(Unknown Source)
    at org.apache.poi.util.DocumentHelper.readDocument(DocumentHelper.java:140)
    at org.apache.poi.POIXMLTypeLoader.parse(POIXMLTypeLoader.java:143)
    at org.openxmlformats.schemas.spreadsheetml.x2006.main.WorksheetDocument$Factory.parse(Unknown Source)
    at org.apache.poi.xssf.usermodel.XSSFSheet.read(XSSFSheet.java:183)
    at org.apache.poi.xssf.usermodel.XSSFSheet.onDocumentRead(XSSFSheet.java:175)
    at org.apache.poi.xssf.usermodel.XSSFWorkbook.parseSheet(XSSFWorkbook.java:438)
    at org.apache.poi.xssf.usermodel.XSSFWorkbook.onDocumentRead(XSSFWorkbook.java:403)
    at org.apache.poi.POIXMLDocument.load(POIXMLDocument.java:190)
    at org.apache.poi.xssf.usermodel.XSSFWorkbook.<init>(XSSFWorkbook.java:266)
    at org.apache.poi.xssf.usermodel.XSSFWorkbook.<init>(XSSFWorkbook.java:336)
    at local_project.banadia_0_1.Banadia.tFileInputExcel_1Process(Banadia.java:1321)
    at local_project.banadia_0_1.Banadia.runJobInTOS(Banadia.java:2828)
    at local_project.banadia_0_1.Banadia.main(Banadia.java:2667)"

[Anonymous]

Hello,

Please try to add the flag to the Job

1. Open the Job in Studio, and navigate to the Advanced > Run tab below the designer view.

2. Click the plus [+] sign next to the JVM parameters window, and a dialog will open where you can add the -XX:-UseGCOverheadLimit flag.

3. Click OK, then save the Job. 

Let us know if it is OK with you.

Best regards

Sabrina

[Anonymous]

Hi Sabrina @xdshi

Thank you for your answer.

I tried what you suggested but I still get an error (I copied it below).
Please, can you think of anything else I can do? This is the configuration of my job:

I know this is unusual, but do you think I could send you the file I'm trying to read to see if the problem is my computer? (it's an excel file .xlsx, 153MB)

Thanks!

May

Starting job Banadia at 11:04 04/01/2019.
[statistics] connecting to socket on port 3374
[statistics] connected
EventID    SourceName    SourcePath    SourceID    ServerName    TicksTimeStamp    EventTimeStamp    EventCategory    Severity    Priority    Message    ConditionName    SubConditionName    AlarmClass    Active    Acked    EffDisabled    Disabled    EddSupressed    Supressed    PersonID    ChangeMask    InputValue    LimitValue    Quality    EventAssociationId    UserComent    UserComputerID    Tag1Value    Tag2Value    Tag3Value    Tag4Value    Shelved    AutoUnshelveTime
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    at com.sun.org.apache.xerces.internal.xni.XMLString.toString(Unknown Source)
    at com.sun.org.apache.xerces.internal.parsers.AbstractDOMParser.characters(Unknown Source)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
    at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source)
    at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source)
    at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(Unknown Source)
    at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(Unknown Source)
    at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(Unknown Source)
    at javax.xml.parsers.DocumentBuilder.parse(Unknown Source)
    at org.apache.poi.util.DocumentHelper.readDocument(DocumentHelper.java:140)
    at org.apache.poi.POIXMLTypeLoader.parse(POIXMLTypeLoader.java:143)
    at org.openxmlformats.schemas.spreadsheetml.x2006.main.WorksheetDocument$Factory.parse(Unknown Source)
    at org.apache.poi.xssf.usermodel.XSSFSheet.read(XSSFSheet.java:183)
    at org.apache.poi.xssf.usermodel.XSSFSheet.onDocumentRead(XSSFSheet.java:175)
    at org.apache.poi.xssf.usermodel.XSSFWorkbook.parseSheet(XSSFWorkbook.java:438)
    at org.apache.poi.xssf.usermodel.XSSFWorkbook.onDocumentRead(XSSFWorkbook.java:403)
    at org.apache.poi.POIXMLDocument.load(POIXMLDocument.java:190)
    at org.apache.poi.xssf.usermodel.XSSFWorkbook.<init>(XSSFWorkbook.java:266)
    at org.apache.poi.xssf.usermodel.XSSFWorkbook.<init>(XSSFWorkbook.java:336)
    at local_project.banadia_0_1.Banadia.tFileInputExcel_1Process(Banadia.java:1321)
    at local_project.banadia_0_1.Banadia.runJobInTOS(Banadia.java:2828)
    at local_project.banadia_0_1.Banadia.main(Banadia.java:2667)

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Thread-0"
[statistics] disconnected

Job Banadia ended at 11:11 04/01/2019. [exit code=1]

[Anonymous]

Hello,

Just to make sure: you are using the 64bit version of Talend Studio as well as a 64 bit version of Java, correct?

Best regards

Sabrina