problem java heap space - taggregate

grimbeorn — Sat, 16 Nov 2024 12:40:21 GMT

Hi all,
I have a problem with a job that begins with a tinputMysql (around 10 mln of rows), checks the latest rows to add, then aggregates (from 11 fileds to 7) with a SUM. I attached some images.
The job starts with a velocity of 18.000 rows/s, then decrease and stop with the error message below:
*********************
Exception in thread "Thread-0" java.lang.OutOfMemoryError: Java heap space
at java.util.LinkedList.listIterator(Unknown Source)
at java.util.AbstractList.listIterator(Unknown Source)
at java.util.AbstractSequentialList.iterator(Unknown Source)
at routines.system.RunStat.sendMessages(RunStat.java:248)
at routines.system.RunStat.run(RunStat.java:212)
at java.lang.Thread.run(Unknown Source)
Exception in thread "Thread-1" java.lang.Error: java.lang.OutOfMemoryError: Java heap space
at mon_sgsn.load_table_fact_blackberry3_0_1.load_table_fact_blackberry3.tMysqlInput_1Process(load_table_fact_blackberry3.java:4672)
at mon_sgsn.load_table_fact_blackberry3_0_1.load_table_fact_blackberry3$1.run(load_table_fact_blackberry3.java:5198)
disconnected
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.util.HashMap.addEntry(Unknown Source)
at java.util.HashMap.put(Unknown Source)
at java.util.ListResourceBundle.loadLookup(Unknown Source)
at java.util.ListResourceBundle.handleGetObject(Unknown Source)
at java.util.ResourceBundle.getObject(Unknown Source)
at java.util.ResourceBundle.getObject(Unknown Source)
at java.util.ResourceBundle.getObject(Unknown Source)
at java.util.ResourceBundle.getStringArray(Unknown Source)
at java.text.DateFormatSymbols.initializeData(Unknown Source)
at java.text.DateFormatSymbols.<init>(Unknown Source)
at java.text.DateFormatSymbols.getInstance(Unknown Source)
at java.text.SimpleDateFormat.<init>(Unknown Source)
at java.text.SimpleDateFormat.<init>(Unknown Source)
at routines.TalendDate.compareDate(TalendDate.java:207)
at mon_sgsn.load_table_fact_blackberry3_0_1.load_table_fact_blackberry3.tMysqlInput_1Process(load_table_fact_blackberry3.java:3651)
... 1 more
*********************

Uploaded with ImageShack.us

Uploaded with ImageShack.us
I tried a lot of configurations oj JVM arguments: Xms128 Xmx768, 256 1024, 128 512...but I got the same problem!
Any suggestion???

Re: problem java heap space - taggregate

Anonymous — Wed, 21 Sep 2011 21:26:19 GMT

The tAggregateRow and tSortRow both suffer from memory management issues when dealing with large data sets. If you are using a MySQL input straight to a tAggregateRow then just do the aggregation within the mysql query. BTW cant see the images, the links send me to a thumbnail.

Re: problem java heap space - taggregate

grimbeorn — Fri, 23 Sep 2011 14:04:47 GMT

sorry for the images, the preview was fine... I try again

What do you mean about doing the aggragation within the query? Changing manually the query within mysql input?

topic Re: problem java heap space - taggregate in Talend Studio

problem java heap space - taggregate

Re: problem java heap space - taggregate

Re: problem java heap space - taggregate