How to handle many jobs for java GC Error(heap size)

Anonymous — Tue, 17 Oct 2017 09:12:59 GMT

Hi. I working on a TO S.

My Job project has lots of Jobs.(below image)

You should look at the left image in a "FRONTDOM_18" Folder, it has 36 Jobs(child job), 2 Starter(Parent job)

and each job is configured below image.

[ setting : Multi thread execution(ON), db connection(shared), db commit(on), db close(off) in the child job)

and starter ( parent job )

parent

[ setting : Multi thread execution(ON), db connection(shared), db close(on) in the child job, Xms 2G, Xmx 4G)

Q, There is no problem when running separately. however, errors occur only when they are bind and executed.

I think It's obviously a problem with memory, but i don't know how deal with this problem.

Is there anything else besides increasing memory?

Q, If solution exists, can i run the Jobs Folder List(same time) on the first image

sorry for bad english skill;

thanks.

Exception in thread "Thread-122" java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.Arrays.copyOf(Unknown Source)
at java.lang.StringCoding.safeTrim(Unknown Source)
at java.lang.StringCoding.access$300(Unknown Source)
at java.lang.StringCoding$StringEncoder.encode(Unknown Source)
at java.lang.StringCoding.encode(Unknown Source)
at java.lang.String.getBytes(Unknown Source)
at org.postgresql.core.Utils.encodeUTF8(Utils.java:54)
at org.postgresql.core.v3.SimpleParameterList.getV3Length(SimpleParameterList.java:234)
at org.postgresql.core.v3.QueryExecutorImpl.sendBind(QueryExecutorImpl.java:808)
at org.postgresql.core.v3.QueryExecutorImpl.sendOneQuery(QueryExecutorImpl.java:1053)
at org.postgresql.core.v3.QueryExecutorImpl.sendQuery(QueryExecutorImpl.java:644)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:345)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeBatch(AbstractJdbc2Statement.java:2674)
at cdc.frontdom_193_198_0_1.FRONTDOM_193_198.tPostgresPlusInput_3Process(FRONTDOM_193_198.java:7841)
at cdc.frontdom_193_198_0_1.FRONTDOM_193_198$4.run(FRONTDOM_193_198.java:15021)
Batch entry 43 <unknown> was aborted. Call getNextException to see the cause.
18_130 : 43
Exception in component tPostgresPlusInput_6 (FRONTDOM_193_198)
org.postgresql.util.PSQLException: Ran out of memory retrieving query results.
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1309)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:192)
at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:451)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:336)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:235)
at cdc.frontdom_193_198_0_1.FRONTDOM_193_198.tPostgresPlusInput_6Process(FRONTDOM_193_198.java:14041)
at cdc.frontdom_193_198_0_1.FRONTDOM_193_198$7.run(FRONTDOM_193_198.java:15153)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
at org.postgresql.core.PGStream.ReceiveTupleV3(PGStream.java:349)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1306)
... 6 more
18_160 : 296
18_192 : 661
18_180 : 17
18_125 : 171


Exception in component tRunJob_273 (FRONTDOM_STARTER_2)
java.lang.RuntimeException: Child job running failed.
org.postgresql.util.PSQLException: Ran out of memory retrieving query results.
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1309)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:192)
at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:451)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:336)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:235)
at cdc.frontdom_193_198_0_1.FRONTDOM_193_198.tPostgresPlusInput_6Process(FRONTDOM_193_198.java:14041)
at cdc.frontdom_193_198_0_1.FRONTDOM_193_198$7.run(FRONTDOM_193_198.java:15153)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
at org.postgresql.core.PGStream.ReceiveTupleV3(PGStream.java:349)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1306)
... 6 more
at cdc.frontdom_starter_2_0_1.FRONTDOM_STARTER_2.tRunJob_273Process(FRONTDOM_STARTER_2.java:4792)
at cdc.frontdom_starter_2_0_1.FRONTDOM_STARTER_2$15.run(FRONTDOM_STARTER_2.java:6436)
18_186 : 532041
FRONTDOM_Stater is completed
521088 milliseconds

Re: How to handle many jobs for java GC Error(heap size)

jilanisyed — Wed, 18 Oct 2017 04:37:14 GMT

please try the Following workaround to solved the problem in Talend

Add new option to Windows–>Preferences–>Talend–>Run/Debug – XX:-UseGCOverheadLimit

Re: How to handle many jobs for java GC Error(heap size)

Anonymous — Mon, 23 Oct 2017 02:44:05 GMT

Thank to reply,

but, I search the error message from my question,

I found it UseGCOverheadLimit option is just error ignore(masking)..

I think preparedstatemts cache size.

Anyway, I solved the problem with Batch Size = 10000 to 5000, exec time little bit slow, but error does not appears,

Thank to reply

topic Re: How to handle many jobs for java GC Error(heap size) in Talend Studio

How to handle many jobs for java GC Error(heap size)

Re: How to handle many jobs for java GC Error(heap size)

Re: How to handle many jobs for java GC Error(heap size)