[resolved] OutOfMemory exception trying to sort delimited file
I am trying to sort a 2.5GB delimited file with 11M rows, 45 columns. tSortRow is configured for external sort. JVM is configured for 2GB. No matter what I try for buffer sizes, I invariably get an OutOfMemory exception like the following:
Exception in thread "Thread-0" java.lang.OutOfMemoryError: Java heap space
at java.util.LinkedList.listIterator(LinkedList.java:667)
at java.util.AbstractList.listIterator(AbstractList.java:284)
at java.util.AbstractSequentialList.iterator(AbstractSequentialList.java:222)
at routines.system.RunStat.sendMessages(RunStat.java:261)
at routines.system.RunStat.run(RunStat.java:225)
at java.lang.Thread.run(Thread.java:662)
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOfRange(Arrays.java:3209)
at java.lang.String.<init>(String.java:215)
at java.lang.StringBuilder.toString(StringBuilder.java:430)
at com.talend.csv.CSVReader.endColumn(CSVReader.java:131)
at com.talend.csv.CSVReader.readNext(CSVReader.java:390)
at johnmdm.sqlinout_0_1.SQLInOut.tFileInputDelimited_1Process(SQLInOut.java:3380)
at johnmdm.sqlinout_0_1.SQLInOut.runJobInTOS(SQLInOut.java:5199)
at johnmdm.sqlinout_0_1.SQLInOut.main(SQLInOut.java:5056)
