topic GC Overhead limit exceeded on server in Talend Studio

GC Overhead limit exceeded on server

Anonymous — Fri, 27 Apr 2018 07:33:29 GMT

Hi,

I am loading around 7.5 million records from a db and after transformation the size of the records doubles to 15 million. But during execution my job got failed with Error "GC Overhead Limit exceeded on server". I am putting xmx - 10240 MB and xms - 1024 MB. Please find my job design below.

tmssqlinput-->tJavaRow-->tExtractJsonFields --> tMap --> tDenormalize --> tJavaRow --> tMSSQLOutput

At tExtractJSONFields data is getting double and at tDenormalize I am merging 2 records to One. I am getting this error at tDenormalize. Is there any better solution to this flow because tDenormalize is holding complete and then passing it one by one.

Best Regards,

Abhishek

Re: GC Overhead limit exceeded on server

Anonymous — Fri, 27 Apr 2018 08:56:04 GMT

Hello,

Could you please try with 4096 or bigger if You have this memory free on Talend Machine?

Best regards

Sabrina

Re: GC Overhead limit exceeded on server

Anonymous — Fri, 27 Apr 2018 09:47:11 GMT

Hi xdishi,
Xmx is already 10240 MB, almost twice as that of 4096.
Or should I increase the xms value ? But I think increasing the memory is not a permanent solution because in our case records count can increase to 40 - 50 millions. In that case we need to again increase the memory.
What I am looking for is a permanent solution. Can we process the records in batches. I mean instead of reading all the records in one go, can we do it in batches.

Best Regards,

Abhishek