topic Re: Java java.lang.OutOfMemoryError Error in Talend Studio

Java java.lang.OutOfMemoryError Error

AndyBrown — Sat, 16 Nov 2024 14:05:37 GMT

I'm getting the following exception while running my job. The job is simple... it takes two inputs (from tPostgreSQLInput components) through a tMap. The result is stored as a delimted TXT file. Where should I be looking to rectify this? Our admin has stated that he has already given the Java VM the max allocation of memory.
### Job STARTED at 2009/01/22 12:13:25 (jobId=lrpusmu3, jobExecutionId=2hSY6uaM) ###
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.lang.String.replace(String.java:1879)
at java.net.URLClassLoader$1.run(URLClassLoader.java:191)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at java.lang.ClassLoader.loadClass(ClassLoader.java:299)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:268)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1309)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:192)
at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:451)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:336)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:235)
at dataextraction.partmaster_0_2.PartMaster.tPostgresqlInput_1Process(PartMaster.java:3361)
at dataextraction.partmaster_0_2.PartMaster.runJobInTOS(PartMaster.java:5611)
at dataextraction.partmaster_0_2.PartMaster.main(PartMaster.java:5491)
### Job ENDED WITH ERROR at 2009/01/22 12:14:51 (jobId=lrpusmu3, jobExecutionId=2hSY6uaM) ###

Re: Java java.lang.OutOfMemoryError Error

Anonymous — Thu, 22 Jan 2009 20:31:20 GMT

you might try using the "Enable stream" option on your main row to your tmap. This can be found under "Advanced settings".
Lookup tables will need to be loaded into memory.

edit for clarification.

Re: Java java.lang.OutOfMemoryError Error

AndyBrown — Thu, 22 Jan 2009 20:42:20 GMT

I may be missing something, but my MAIN row tPostgreSQLInput component only has the settings: "Use Cursor", "Trim all the String/Char columns", and "tStatCatcher Statistics" options under Advanced Options. I'm using version 2.4.1. Was the Enable Stream option introduced after that?

Re: Java java.lang.OutOfMemoryError Error

Anonymous — Thu, 22 Jan 2009 22:57:25 GMT

"Enable stream" is the MySQL equivalent of Postgres "Enable Cursor"
Checking this option in your component should help you to reduce memory consuption on the main input.
This is not the only solution but it can be done as a first try.

Inyour lookup do you load all the columns of your table or did you select only the really required columns ?
HTH,

Re: Java java.lang.OutOfMemoryError Error

AndyBrown — Fri, 23 Jan 2009 16:23:21 GMT

Ok. On my MAIN row tPostgreSQLInput, I checked the "Use Cursor" option and left the value at "1000". I'm still getting the same exception.

Re: Java java.lang.OutOfMemoryError Error

AndyBrown — Thu, 29 Jan 2009 23:37:35 GMT

What would the other things I can try to eliminate this exception? Here's what we've done thusfar:
1) Allocated 2GB of memory to the Java VM
2) Checked the "Use Cursor" option (with a setting of 10,000) on the largest database input table (which was the MAIN row into the tMap)
Thanks.

Re: Java java.lang.OutOfMemoryError Error

Anonymous — Thu, 29 Jan 2009 23:54:57 GMT

What's the settings in your .ini file?
-vmargs
-Xms40m
-Xmx500m
-XX:MaxPermSize=128m
Bluesin

Re: Java java.lang.OutOfMemoryError Error

AndyBrown — Fri, 30 Jan 2009 17:36:01 GMT

The settings are exactly as you show...
-vmargs
-Xms40m
-Xmx500m
-XX:MaxPermSize=128m
The file is TISPE-linux-gtk-x86.ini
Thanks.

Re: Java java.lang.OutOfMemoryError Error

Anonymous — Fri, 30 Jan 2009 17:54:17 GMT

you might try tweaking the Xmx500m value to a little higher.
Also, removing unused columns from the schema can help reduce memory consumption.

Re: Java java.lang.OutOfMemoryError Error

AndyBrown — Fri, 30 Jan 2009 21:50:22 GMT

It looks like bumping the config and specifying the exact columns I needed (instead of SELECT *) did the trick. Thanks.