Do not input private or sensitive data. View Qlik Privacy & Cookie Policy.
Skip to main content

cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Announcements
See why IDC MarketScape names Qlik a 2025 Leader! Read more
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Anonymous
Not applicable
‎2012-07-13 06:38 AM

OutofMemory Error When considering a huge XML File as source

Hi Shong,
I have a SDMX XML File as my source and earlier xml file size was in kb and able to run the jobs successfully.
Now we got a very huge 1GB XMl file and if run the same job job getting failed and throwing the below error.
Can you please suggest me with proper workarounds and their steps.
Please help me as it is very urgent and and life issue.
Error 1:
===========
Starting job j500_ValidateLoad at 14:51 12/07/2012.
connecting to socket on port 3958
connected
Exception in thread "Thread-7" java.lang.OutOfMemoryError: Java heap space
at java.util.LinkedList.listIterator(LinkedList.java:667)
at java.util.AbstractList.listIterator(AbstractList.java:284)
at java.util.AbstractSequentialList.iterator(AbstractSequentialList.java:222)
at routines.system.RunStat.sendMessages(RunStat.java:248)
at routines.system.RunStat.run(RunStat.java:212)
at java.lang.Thread.run(Thread.java:619)
Exception in thread "main" java.lang.Error: java.lang.Error: java.lang.Error: java.lang.Error: java.lang.Error: java.lang.OutOfMemoryError: Java heap space
at datasetimportjobs.j500_validateload_6_0.j500_ValidateLoad.tOracleInput_1Process(j500_ValidateLoad.java:4042)
at datasetimportjobs.j500_validateload_6_0.j500_ValidateLoad.runJobInTOS(j500_ValidateLoad.java:5436)
at datasetimportjobs.j500_validateload_6_0.j500_ValidateLoad.main(j500_ValidateLoad.java:5113)
Caused by: java.lang.Error: java.lang.Error: java.lang.Error: java.lang.Error: java.lang.OutOfMemoryError: Java heap space
disconnected
at datasetimportjobs.j100_read_files_6_1.j100_Read_Files.tRunJob_1Process(j100_Read_Files.java:2555)
at datasetimportjobs.j100_read_files_6_1.j100_Read_Files.runJobInTOS(j100_Read_Files.java:2942)
at datasetimportjobs.j100_read_files_6_1.j100_Read_Files.runJob(j100_Read_Files.java:2608)
at datasetimportjobs.j500_validateload_6_0.j500_ValidateLoad.tOracleInput_1Process(j500_ValidateLoad.java:3375)
... 2 more
Caused by: java.lang.Error: java.lang.Error: java.lang.Error: java.lang.OutOfMemoryError: Java heap space
at datasetimportjobs.j100_read_files_6_1.j100_Read_Files.tRunJob_2Process(j100_Read_Files.java:956)
at datasetimportjobs.j100_read_files_6_1.j100_Read_Files.tRunJob_1Process(j100_Read_Files.java:2546)
... 5 more
Caused by: java.lang.Error: java.lang.Error: java.lang.OutOfMemoryError: Java heap space
at datasetimportjobs.j102_read_dataset_xml_0_6.j102_Read_Dataset_XML.tFileList_1Process(j102_Read_Dataset_XML.java:1030)
at datasetimportjobs.j102_read_dataset_xml_0_6.j102_Read_Dataset_XML.runJobInTOS(j102_Read_Dataset_XML.java:1498)
at datasetimportjobs.j102_read_dataset_xml_0_6.j102_Read_Dataset_XML.runJob(j102_Read_Dataset_XML.java:1282)
at datasetimportjobs.j100_read_files_6_1.j100_Read_Files.tRunJob_2Process(j100_Read_Files.java:895)
... 6 more
Caused by: java.lang.Error: java.lang.OutOfMemoryError: Java heap space
at datasetimportjobs.j140_import_segments_0_6.j140_Import_Segments.tFileInputXML_1Process(j140_Import_Segments.java:2345)
at datasetimportjobs.j140_import_segments_0_6.j140_Import_Segments.runJobInTOS(j140_Import_Segments.java:2767)
at datasetimportjobs.j140_import_segments_0_6.j140_Import_Segments.runJob(j140_Import_Segments.java:2572)
at datasetimportjobs.j102_read_dataset_xml_0_6.j102_Read_Dataset_XML.tFileList_1Process(j102_Read_Dataset_XML.java:762)
... 9 more
Caused by: java.lang.OutOfMemoryError: Java heap space
at org.dom4j.DocumentFactory.createAttribute(DocumentFactory.java:156)
at org.dom4j.tree.AbstractElement.setAttributes(AbstractElement.java:549)
at org.dom4j.io.SAXContentHandler.addAttributes(SAXContentHandler.java:916)
at org.dom4j.io.SAXContentHandler.startElement(SAXContentHandler.java:249)
at org.apache.xerces.parsers.AbstractSAXParser.startElement(Unknown Source)
at org.apache.xerces.parsers.AbstractXMLDocumentParser.emptyElement(Unknown Source)
at org.apache.xerces.impl.XMLNSDocumentScannerImpl.scanStartElement(Unknown Source)
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source)
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)
at org.dom4j.io.SAXReader.read(SAXReader.java:465)
at datasetimportjobs.j140_import_segments_0_6.j140_Import_Segments.tFileInputXML_1Process(j140_Import_Segments.java:1601)
... 12 more
Job j500_ValidateLoad ended at 14:58 12/07/2012.
Error 2:
==========
Starting job j500_ValidateLoad at 15:23 12/07/2012.
connecting to socket on port 3527
connected
Exception in thread "main" java.lang.Error: java.lang.Error: java.lang.Error: java.lang.Error: java.lang.Error: java.lang.OutOfMemoryError: Java heap space
at datasetimportjobs.j500_validateload_6_0.j500_ValidateLoad.tOracleInput_1Process(j500_ValidateLoad.java:4042)
at datasetimportjobs.j500_validateload_6_0.j500_ValidateLoad.runJobInTOS(j500_ValidateLoad.java:5436)
at datasetimportjobs.j500_validateload_6_0.j500_ValidateLoad.main(j500_ValidateLoad.java:5113)
Caused by: java.lang.Error: java.lang.Error: java.lang.Error: java.lang.Error: java.lang.OutOfMemoryError: Java heap space
disconnected
at datasetimportjobs.j100_read_files_6_1.j100_Read_Files.tRunJob_1Process(j100_Read_Files.java:2555)
at datasetimportjobs.j100_read_files_6_1.j100_Read_Files.runJobInTOS(j100_Read_Files.java:2942)
at datasetimportjobs.j100_read_files_6_1.j100_Read_Files.runJob(j100_Read_Files.java:2608)
at datasetimportjobs.j500_validateload_6_0.j500_ValidateLoad.tOracleInput_1Process(j500_ValidateLoad.java:3375)
... 2 more
Caused by: java.lang.Error: java.lang.Error: java.lang.Error: java.lang.OutOfMemoryError: Java heap space
at datasetimportjobs.j100_read_files_6_1.j100_Read_Files.tRunJob_2Process(j100_Read_Files.java:956)
at datasetimportjobs.j100_read_files_6_1.j100_Read_Files.tRunJob_1Process(j100_Read_Files.java:2546)
... 5 more
Caused by: java.lang.Error: java.lang.Error: java.lang.OutOfMemoryError: Java heap space
at datasetimportjobs.j102_read_dataset_xml_0_6.j102_Read_Dataset_XML.tFileList_1Process(j102_Read_Dataset_XML.java:1030)
at datasetimportjobs.j102_read_dataset_xml_0_6.j102_Read_Dataset_XML.runJobInTOS(j102_Read_Dataset_XML.java:1498)
at datasetimportjobs.j102_read_dataset_xml_0_6.j102_Read_Dataset_XML.runJob(j102_Read_Dataset_XML.java:1282)
at datasetimportjobs.j100_read_files_6_1.j100_Read_Files.tRunJob_2Process(j100_Read_Files.java:895)
... 6 more
Caused by: java.lang.Error: java.lang.OutOfMemoryError: Java heap space
at datasetimportjobs.j140_import_segments_0_6.j140_Import_Segments.tFileInputXML_1Process(j140_Import_Segments.java:2345)
at datasetimportjobs.j140_import_segments_0_6.j140_Import_Segments.runJobInTOS(j140_Import_Segments.java:2767)
at datasetimportjobs.j140_import_segments_0_6.j140_Import_Segments.runJob(j140_Import_Segments.java:2572)
at datasetimportjobs.j102_read_dataset_xml_0_6.j102_Read_Dataset_XML.tFileList_1Process(j102_Read_Dataset_XML.java:762)
... 9 more
Caused by: java.lang.OutOfMemoryError: Java heap space
at org.dom4j.DocumentFactory.createAttribute(DocumentFactory.java:156)
at org.dom4j.tree.AbstractElement.setAttributes(AbstractElement.java:549)
at org.dom4j.io.SAXContentHandler.addAttributes(SAXContentHandler.java:916)
at org.dom4j.io.SAXContentHandler.startElement(SAXContentHandler.java:249)
at org.apache.xerces.parsers.AbstractSAXParser.startElement(Unknown Source)
at org.apache.xerces.parsers.AbstractXMLDocumentParser.emptyElement(Unknown Source)
at org.apache.xerces.impl.XMLNSDocumentScannerImpl.scanStartElement(Unknown Source)
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source)
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)
at org.dom4j.io.SAXReader.read(SAXReader.java:465)
at datasetimportjobs.j140_import_segments_0_6.j140_Import_Segments.tFileInputXML_1Process(j140_Import_Segments.java:1601)
... 12 more
Exception in thread "Thread-7" java.lang.OutOfMemoryError: Java heap space
at java.util.LinkedList.listIterator(LinkedList.java:667)
at java.util.AbstractList.listIterator(AbstractList.java:284)
at java.util.AbstractSequentialList.iterator(AbstractSequentialList.java:222)
at routines.system.RunStat.sendMessages(RunStat.java:248)
at routines.system.RunStat.run(RunStat.java:212)
at java.lang.Thread.run(Thread.java:619)
Job j500_ValidateLoad ended at 15:25 12/07/2012.
Thanks,
KPK
Labels (4)
Labels
775 Views
0 Likes
15 Replies
Anonymous
Not applicable
‎2012-07-17 05:56 AM
Author

This didn't Work for me...............Still iam getting the same Error
207 Views
0 Likes
Anonymous
Not applicable
‎2012-07-18 01:12 PM
Author

Any Help?????????
207 Views
0 Likes
Anonymous
Not applicable
‎2012-08-03 05:30 AM
Author

I have 300 MB XML File and as suggested changed all the JVM arguments to 256/512/1024 and 1024,using SAX Parser and Still iam facing the same issue..can anyone please help me out with some workarounds!!
207 Views
0 Likes
Anonymous
Not applicable
‎2012-08-03 05:36 AM
Author

Hi
So there is the component tRunjob in your parent job?
You might use the feature of tRunJob--Use an independent process to run subjob.
It will reduce the memory consuming. But make sure there is no context or data transmition between parent job and this subjob. Or you will get error.
Regards,
Pedro
207 Views
0 Likes
Anonymous
Not applicable
‎2012-08-03 05:54 AM
Author

Pedro,
Context is there and data transmission happends.....
What will be the reason behind this......it is not able to handle 300MB file...then how it will handle large files of size 5 or 10GB???

Thanks,
KPK
207 Views
0 Likes
Anonymous
Not applicable
‎2012-08-03 05:59 AM
Author

Hi KPK
All the parent job and the child jobs share only one JVM thread.
The maximum memory is the JVM argument xmx.
Try to increase the JVM arguments. If it doesn't work either you have to simplify your jobs.
The feature "Use an independent process to run subjob" will create a new JVM thread for the subjob. But you say you can't because you need to transmit context variables.
Regards,
Pedro
207 Views
0 Likes