topic Re: GC out of memory error in Talend Studio

GC out of memory error

Anonymous — Fri, 29 Mar 2019 14:36:08 GMT

Hi,

I created multiple outputs and after run I got out of memory error. Actually I have configured my JVM Argument to 1024-9162 . I hope I can find permanent solution for this error. Another thing is when I run multiple outputs in one job it will be always super slow.

Thanks

Re: GC out of memory error

Anonymous — Fri, 29 Mar 2019 15:01:07 GMT

@szhou1

Make sure you go into the Run Window in the Designer View and change your Advance Settings (JVM Settings)based on your memory configuration on your machine. Default is -Xms256M -Xmx1024M. I have 8GB of RAM on my machine so my settings are in the ss below.

Re: GC out of memory error

Anonymous — Fri, 29 Mar 2019 15:05:21 GMT

You can find the right values on line if you search for JVM Settings for Talend based on Memory. You can also Split the Input File into multiple files. If you're using a Lookup make sure you're only bringing in the Lookup values you need. Make sure you filter any input data before bringing it into a tMap component.

You can store data into a tHashOutput ( in-memory table) if you're lookup is quite large.

Let me know if this helps

Andrew

Re: GC out of memory error

Anonymous — Fri, 29 Mar 2019 15:10:32 GMT

@szhou1 May i know how many rows you have in the source, and how many in the look ups ?

Re: GC out of memory error

JBristow — Fri, 29 Mar 2019 15:39:46 GMT

The issue can be addressed as many have responded with JVM settings appropriate for the job. This can be done in the job itself while running in the Studio by selecting the "Run" tab and then applying the appropriate memory allocation - and you can always apply those settings in the TAC when deploying the job.

I've also had to consider memory issues when designing my data flow - based on volume of data being processed. I have one job that I was forced to split the input file into smaller units as a primary step - and then iterate through those smaller units.

So basically - there's no "one solution" to this - because it depends on a number of factors; but you have options to manage it.

Good luck!

Re: GC out of memory error

Anonymous — Fri, 29 Mar 2019 15:44:41 GMT

Main is around 200,000.
Lookup is around 400,000

Thanks

Re: GC out of memory error

Anonymous — Fri, 29 Mar 2019 15:51:05 GMT

1) verify the RAM in the TAC server it is important, as while running on TAC you may run multiple jobs so, not to get effected, check it

2)Another option is at tMap you can generate intermediate files like below link you can follow form the file as input

https://community.talend.com/t5/Design-and-Development/GC-overhead-limit-error-when-running-Jobs-in-Studio/m-p/153212?author_id=19335&collapse_discussion=true&filter=location,authorId&location=forum-board:integrating&q=Gc&search_type=thread

3) if you use tHashcomponent it can consume more memory with in the job

this memory isssues can come in many formats.