topic OutOfMemoryError: Java heap space in Talend Studio

OutOfMemoryError: Java heap space

Anonymous — Mon, 06 May 2013 10:30:10 GMT

Hello,
At the creation of an XML file (tAdvancedFileOutputXML), we had a problem: java.lang.OutOfMemoryError: Java heap space.
We solved by changing the parameter: generation mode "Fast memory-consuming goal - Dom4J" replaced by "Slow with no memory Consumed"
Can you explain how this generation mode work?
Is there a maximum size of the XML file?
Thanks and Regards.

Re: OutOfMemoryError: Java heap space

Anonymous — Mon, 06 May 2013 11:05:50 GMT

Hi,
Here is a component reference tAdvancedFileOutputXML.
For the "OutOfMemoryErrorJava heap space issue", there are also workaround outOfMemory and Allocating more memory to Talend Studio
Best regards
Sabrina

Re: OutOfMemoryError: Java heap space

Anonymous — Mon, 06 May 2013 13:55:38 GMT

Is there a maximum size of the XML file?
I'll have to deal with files becoming larger and I would anticipate a blocking
Best regards.

Re: OutOfMemoryError: Java heap space

Anonymous — Mon, 06 May 2013 16:25:31 GMT

Hi Noel,
in my experience, you will start having heap and memory problems like GC overhead limit exceeded with xml files larger than 700Mb even after hacking VM arguments and using SAX;
i had to deal with 1.2 up to 2Gb xml files as inputs, and the only way to do it was to split files into smaller one like 6Mb each and to a tfilelist to process them using the Dom4J parser.
user brazabr helped me with this useful code that i added as a routine and used it in a tjava componement.

    public static boolean split_file(String filename, int maxpart, String tagname, String roottag, String nsdeclaration){
        FileOutputStream fout = null;    
        PrintStream outstream = null;
        Scanner s = null;
        int part=0;
        int partsize=0;
        boolean partnew=true;
        String partfile, suffix, token;
        partfile = filename.replaceFirst("\\.xml$", "");
        try {
            s = new Scanner(new FileInputStream(filename),"utf-8");
            s.useDelimiter("</" + tagname + ">");
            while (s.hasNext()) {
                if(partnew){ //begin a new part file
                    suffix = String.format("_part%04d.xml",part);
                    fout = new FileOutputStream (partfile + suffix);
                    outstream = new PrintStream(fout);
                    if (part>0){ //insert leading tags
                        outstream.println("<?xml version=\"1.0\" encoding=\"utf-8\"?>");
                        outstream.println("<" + roottag + " " + nsdeclaration + ">");
                    }
                    partsize=0;
                    partnew=false;
                }
                //just append tokens
                token = s.next();
                outstream.print(token);
                //if not last chunk append closing tag
                if (token.indexOf("</" + roottag + ">")<0)     outstream.println("</" + tagname + ">");
                partsize += token.length();
                if (partsize > maxpart) { //time to wrap it up
                    outstream.println("</" + roottag + ">");
                    outstream.close();
                    outstream = null;
                    fout.close();
                    fout = null;
                    part++;
                    partnew = true;
                }
            }
            //dump the remaining part to out
            outstream.close();
            //fout.close();
            return true;
        } catch (Exception e) {
            System.out.println(e.getMessage());
            if (s != null) {
                s.close();
            }
            if (outstream != null) {
                outstream.close();
            }
            return false;
        }
    }

Re: OutOfMemoryError: Java heap space

Anonymous — Tue, 03 Sep 2013 16:12:34 GMT

Hello,
I have seen lot of people having issue with handling Huge XML files.
I have written a small post for working with Large XML files in Talend. Please visit the link below for more details:
http://www.vikramtakkar.com/2013/09/handling-huge-xml-files-in-talend.html
Let me know, if it helps.

Re: OutOfMemoryError: Java heap space

Anonymous — Tue, 03 Sep 2013 16:49:31 GMT

hi all,
there is no determinated size to say that it will be out of heap (of number of java object).
I read a file about 5Go (with SAX) but only several elements.
So it could be empiric to know when a job crash due to too many java object.
BUT you have to optimize your job (use write on disk option for tmap, sort, ect) , avoid using live memory with buffer & hash component, read in several time, increase jvm params
my 2cents
regards
laurent