topic Re: java.lang.OutOfMemoryError: GC overhead limit exceeded while migration high volume data in Talend Studio

java.lang.OutOfMemoryError: GC overhead limit exceeded while migration high volume data

Anonymous — Sat, 16 Nov 2024 03:39:06 GMT

Hi Experts,

We are doing a migration from oracle to postgre using talend 6.4. It's observed that java.lang.OutOfMemoryError: GC overhead limit exceeded is thrown when the volume increases. We currently had heap size set to 12 GB which isnot enough for 18 million records, when the heap was increased to 15 gb job completed. We have around 100 million records to be migrated in production , request you to help optimize memory utilization and any known hacks.

Found some documentation to use disk to store data processed via tmap instead of memory(https://help.talend.com/reader/EJfmjmfWqXUp5sadUwoGBA/J4xg5kxhK1afr7i7rFA65w) but i am not able to find the settings to enable store temp data as true(https://help.talend.com/viewer/attachment/j7gt4x19HWatE4J4vdPsKQ).

We also tried removing the tmap still there is memory issue. The input of the job is a view connecting multiple tables. we have multiple sub jobs for different high volume tables and it's observed that memory is not getting released after processing high volume data jobs.

Thank you in advance,

Rajeswary S

Re: java.lang.OutOfMemoryError: GC overhead limit exceeded while migration high volume data

Anonymous — Fri, 10 Jan 2020 09:08:54 GMT

To enable store temp data as true, configure it in the lookup table settings, see

and then select a temp data directory.

Enable to store temp data on disk on other memory consumption component such as tSortRow. It is better to share a screenshot of your job design, we will see if it is possible to optimize the job.

Regards

Shong

Re: java.lang.OutOfMemoryError: GC overhead limit exceeded while migration high volume data

Anonymous — Fri, 10 Jan 2020 11:33:11 GMT

settingsnotavailable

Hi Shong,

Thank you for the reply, we do saw an option mentioned in a link, but it is not visible in Talend studio 6.4 version which we are using .

Warm Regards,

Rajeswary S

Re: java.lang.OutOfMemoryError: GC overhead limit exceeded while migration high volume data

Anonymous — Tue, 04 Feb 2020 00:01:09 GMT

Hi Shong,

Kindly advise on how to proceed, we are blocked.

Warm Regards,

Rajeswary S

Re: java.lang.OutOfMemoryError: GC overhead limit exceeded while migration high volume data

Anonymous — Tue, 04 Feb 2020 07:18:11 GMT

Hi
The 'Store temp data on disk' option is in the settings panel of lookup table rather than main table.

Regards
Shong

Re: java.lang.OutOfMemoryError: GC overhead limit exceeded while migration high volume data

David_Beaty — Tue, 04 Feb 2020 07:44:21 GMT

Hi,

Aside from the suggestions from @shong, which should work, for components like tMap, tSortRow and tUniqRow (to mention a few), also consider whether migrating 100Million rows in one go is the best approach. Is there some form of selectable key like a date range or some other blocking key that allows you to break the data up into more manageable chunks and migrate a chunk at a time.

This would also improve the recoverability of the job if it should have an issue part way through the migration.