java.lang.OutOfMemoryError: GC overhead limit exceeded

[Anonymous]

Hi,

I got out of memory error when I run my job. my inputs have over 1 million data and I have set JVM setting to Xms1024m-Xmx8192m but it still shows out of memory.

Can you please advise me how I can deal with this error?

Thanks a lot!

[Anonymous]

Hello,

At a first step, could you please let us know if this article helps for you?

https://community.talend.com/t5/Migration-Configuration-and/GC-overhead-limit-error-when-running-Job...

Best regards

Sabrina

[Anonymous]

Hi,

I added a flag as the article said but I still got "out of memory" error. 

One source Workbook has more than 1million data. Another one ISV_US has about 0.5 million data

[Anonymous]

Hello,

With your tMap component, are you using temp directory storage to improve the performance?

For a large set of data, try to store the data on disk instead of memory.

Best regards

Sabrina

 

[David_Beaty]

Hi,

1/ Use Temp data store as xdshi suggests

2/ In the lookup config of the tMap (spanner icon on the lookup flow in tMap), set "Store Temp Data" to true

3/ Consider swapping the main/lookup around so the smaller dataset is the lookup and larger is the main.

Thanks

David