problem java heap space - taggregate

[grimbeorn]

Hi all,
I have a problem with a job that begins with a tinputMysql (around 10 mln of rows), checks the latest rows to add, then aggregates (from 11 fileds to 7) with a SUM. I attached some images.
The job starts with a velocity of 18.000 rows/s, then decrease and stop with the error message below:
*********************
Exception in thread "Thread-0" java.lang.OutOfMemoryError: Java heap space
at java.util.LinkedList.listIterator(Unknown Source)
at java.util.AbstractList.listIterator(Unknown Source)
at java.util.AbstractSequentialList.iterator(Unknown Source)
at routines.system.RunStat.sendMessages(RunStat.java:248)
at routines.system.RunStat.run(RunStat.java:212)
at java.lang.Thread.run(Unknown Source)
Exception in thread "Thread-1" java.lang.Error: java.lang.OutOfMemoryError: Java heap space
at mon_sgsn.load_table_fact_blackberry3_0_1.load_table_fact_blackberry3.tMysqlInput_1Process(load_table_fact_blackberry3.java:4672)
at mon_sgsn.load_table_fact_blackberry3_0_1.load_table_fact_blackberry3$1.run(load_table_fact_blackberry3.java:5198)
disconnected
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.util.HashMap.addEntry(Unknown Source)
at java.util.HashMap.put(Unknown Source)
at java.util.ListResourceBundle.loadLookup(Unknown Source)
at java.util.ListResourceBundle.handleGetObject(Unknown Source)
at java.util.ResourceBundle.getObject(Unknown Source)
at java.util.ResourceBundle.getObject(Unknown Source)
at java.util.ResourceBundle.getObject(Unknown Source)
at java.util.ResourceBundle.getStringArray(Unknown Source)
at java.text.DateFormatSymbols.initializeData(Unknown Source)
at java.text.DateFormatSymbols.<init>(Unknown Source)
at java.text.DateFormatSymbols.getInstance(Unknown Source)
at java.text.SimpleDateFormat.<init>(Unknown Source)
at java.text.SimpleDateFormat.<init>(Unknown Source)
at routines.TalendDate.compareDate(TalendDate.java:207)
at mon_sgsn.load_table_fact_blackberry3_0_1.load_table_fact_blackberry3.tMysqlInput_1Process(load_table_fact_blackberry3.java:3651)
... 1 more
*********************

Uploaded with ImageShack.us

Uploaded with ImageShack.us
I tried a lot of configurations oj JVM arguments: Xms128 Xmx768, 256 1024, 128 512...but I got the same problem!
Any suggestion???

[Anonymous]

The tAggregateRow and tSortRow both suffer from memory management issues when dealing with large data sets. If you are using a MySQL input straight to a tAggregateRow then just do the aggregation within the mysql query. BTW cant see the images, the links send me to a thumbnail.

[grimbeorn]

sorry for the images, the preview was fine... I try again


What do you mean about doing the aggragation within the query? Changing manually the query within mysql input?