topic Re: java.lang.OutOfMemoryError: GC overhead limit exceeded in Talend Studio

java.lang.OutOfMemoryError: GC overhead limit exceeded

Anonymous — Thu, 18 Oct 2018 21:27:45 GMT

Hi,

I got out of memory error when I run my job. my inputs have over 1 million data and I have set JVM setting to Xms1024m-Xmx8192m but it still shows out of memory.

Can you please advise me how I can deal with this error?

Thanks a lot!

Re: java.lang.OutOfMemoryError: GC overhead limit exceeded

Anonymous — Fri, 19 Oct 2018 04:03:26 GMT

Hello,

At a first step, could you please let us know if this article helps for you?

https://community.talend.com/t5/Migration-Configuration-and/GC-overhead-limit-error-when-running-Jobs-in-Studio/ta-p/40280

Best regards

Sabrina

Re: java.lang.OutOfMemoryError: GC overhead limit exceeded

Anonymous — Fri, 19 Oct 2018 15:28:33 GMT

Hi,

I added a flag as the article said but I still got "out of memory" error.

One source Workbook has more than 1million data. Another one ISV_US has about 0.5 million data

Re: java.lang.OutOfMemoryError: GC overhead limit exceeded

Anonymous — Wed, 31 Oct 2018 08:26:09 GMT

Hello,

With your tMap component, are you using temp directory storage to improve the performance?

For a large set of data, try to store the data on disk instead of memory.

Best regards

Sabrina

Re: java.lang.OutOfMemoryError: GC overhead limit exceeded

David_Beaty — Wed, 31 Oct 2018 08:37:19 GMT

Hi,

1/ Use Temp data store as xdshi suggests

2/ In the lookup config of the tMap (spanner icon on the lookup flow in tMap), set "Store Temp Data" to true

3/ Consider swapping the main/lookup around so the smaller dataset is the lookup and larger is the main.

Thanks

David