topic Re: [resolved] OutOfMemoryError: GC overhead limit exceeded in simple MSSqBulkExec jb in Talend Studio

[resolved] OutOfMemoryError: GC overhead limit exceeded in simple MSSqBulkExec jb

Anonymous — Sat, 16 Nov 2024 12:06:33 GMT

I'm doing a simple, three component job - tFileInputExcel > tMap > MSSqBulkOutputExec.
The input file has just 11,923 rows (was just written by another TOSDI job) and the tMap has no processing except some row mapping.
The MSSqBulkOutputExec uses a just-retrieved repository definition for the database table and is set to append to the SQL table.
Execution starts out slow and slows to a crawl until it stops at row 5,287 with the following: (The temporary file, mssql_data.txt finishes with 5,206 rows written.)
Exception in thread "main" java.lang.Error: java.lang.OutOfMemoryError: GC overhead limit exceeded
at masterproviderdatabase.dhpl_all_insertlicenseesintompdproviders_0_1.DHPL_All_InsertLicenseesIntoMPDProviders.tFileInputExcel_1Process(DHPL_All_InsertLicenseesIntoMPDProviders.java:3830)
at masterproviderdatabase.dhpl_all_insertlicenseesintompdproviders_0_1.DHPL_All_InsertLicenseesIntoMPDProviders.runJobInTOS(DHPL_All_InsertLicenseesIntoMPDProviders.java:4012)
at masterproviderdatabase.dhpl_all_insertlicenseesintompdproviders_0_1.DHPL_All_InsertLicenseesIntoMPDProviders.main(DHPL_All_InsertLicenseesIntoMPDProviders.java:3877)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
at sun.reflect.GeneratedConstructorAccessor6.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
at org.apache.xmlbeans.impl.schema.SchemaTypeImpl.createUnattachedNode(SchemaTypeImpl.java:1859)
disconnected
at org.apache.xmlbeans.impl.schema.SchemaTypeImpl.createElementType(SchemaTypeImpl.java:1021)
at org.apache.xmlbeans.impl.values.XmlObjectBase.create_element_user(XmlObjectBase.java:893)
at org.apache.xmlbeans.impl.store.Xobj.getUser(Xobj.java:1657)
at org.apache.xmlbeans.impl.store.Xobj.find_element_user(Xobj.java:2062)
at org.openxmlformats.schemas.spreadsheetml.x2006.main.impl.CTCellImpl.getIs(Unknown Source)
at org.apache.poi.xssf.usermodel.XSSFCell.getRichStringCellValue(XSSFCell.java:269)
at org.apache.poi.xssf.usermodel.XSSFCell.getRichStringCellValue(XSSFCell.java:64)
at masterproviderdatabase.dhpl_all_insertlicenseesintompdproviders_0_1.DHPL_All_InsertLicenseesIntoMPDProviders.tFileInputExcel_1Process(DHPL_All_InsertLicenseesIntoMPDProviders.java:2457)
... 2 more
I've done much bigger SQL outputs that this. I tied restarting Windows(7) to clear any cobwebs but no difference.
Any suggestions? Thanks!
UPDATE: I just replaced the MSSqBulkOutputExec then the tMap with a tLogRow, disconnecting them until I only have the FileInputExcel and the tLogRow - it still fails the same! ??
UPDATE: On the Advanced Settings tab of the FileInputExcel I found the Generation Mode field and set it to "Less memory consumed . . . " All set now.

Re: [resolved] OutOfMemoryError: GC overhead limit exceeded in simple MSSqBulkExec jb

Anonymous — Thu, 07 Feb 2013 02:31:57 GMT

Hi,
Would you mind sharing your resolution with us? All of us are interested with it, thanks a lot.
Best regards
Sabrina

Re: [resolved] OutOfMemoryError: GC overhead limit exceeded in simple MSSqBulkExec jb

soujanyam — Fri, 22 Aug 2014 11:45:18 GMT

Hi everyone,
I'm also facing same problem. When my input files increases at tFilelist in my job I'm getting error as
Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
    at java.io.BufferedReader.<init>(BufferedReader.java:98)
    at java.io.BufferedReader.<init>(BufferedReader.java:109)
    at org.talend.fileprocess.TOSDelimitedReader.init(TOSDelimitedReader.java:109)
    at org.talend.fileprocess.TOSDelimitedReader.<init>(TOSDelimitedReader.java:90)
    at org.talend.fileprocess.FileInputDelimited.<init>(FileInputDelimited.java:167)
    at sample.rech_profile_0_1.Rech_profile.tFileList_4Process(Rech_profile.java:6771)
    at sample.rech_profile_0_1.Rech_profile.tHashInput_tUnite_1Process(Rech_profile.java:16932)
    at sample.rech_profile_0_1.Rech_profile.runJobInTOS(Rech_profile.java:18070)
    at sample.rech_profile_0_1.Rech_profile.main(Rech_profile.java:17953)
What should I do?Please provide the solution to me.Any help would be appreciated.
Thanks in advance.

Re: [resolved] OutOfMemoryError: GC overhead limit exceeded in simple MSSqBulkExec jb

Anonymous — Fri, 22 Aug 2014 11:49:41 GMT

Try below option
Add new option to Windows-->Preferences-->Talend-->Run/Debug - XX:-UseGCOverheadLimit
https://community.talend.com/t5/Design-and-Development/resolved-Solution-for-quot-java-lang-OutOfMemoryError-GC/td-p/113787
Thanks
Vaibhav

Re: [resolved] OutOfMemoryError: GC overhead limit exceeded in simple MSSqBulkExec jb

soujanyam — Fri, 22 Aug 2014 13:13:51 GMT

Thanks for your prompt reply vaibav.
I'll check it up right now. I've send an mail to you for your help.

Re: [resolved] OutOfMemoryError: GC overhead limit exceeded in simple MSSqBulkExec jb

Anonymous — Fri, 22 Aug 2014 13:32:17 GMT

hi all,
be aware that JVM param doesn't correct the reason why you've got probably a too smal heap for your app.
you just disable the limit of 98% accepted for GC. It could solve your problem but for a while.
http://javaeesupportpatterns.blogspot.fr/2011/08/gc-overhead-limit-exceeded-problem-and.html
regards
laurent

Re: [resolved] OutOfMemoryError: GC overhead limit exceeded in simple MSSqBulkExec jb

soujanyam — Fri, 22 Aug 2014 14:00:06 GMT

Ya I tried as Vaibav said
But got error again like
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space

at sample.rech_profile_0_1.Rech_profile.tFileList_4Process(Rech_profile.java:7344)
at sample.rech_profile_0_1.Rech_profile.tHashInput_tUnite_1Process(Rech_profile.java:16932)
at sample.rech_profile_0_1.Rech_profile.runJobInTOS(Rech_profile.java:18070)
at sample.rech_profile_0_1.Rech_profile.main(Rech_profile.java:17953)

Re: [resolved] OutOfMemoryError: GC overhead limit exceeded in simple MSSqBulkExec jb

soujanyam — Fri, 22 Aug 2014 14:04:27 GMT

Thanks for your prompt reply laurent,
But provide the solution to me.

Re: [resolved] OutOfMemoryError: GC overhead limit exceeded in simple MSSqBulkExec jb

Anonymous — Fri, 22 Aug 2014 14:23:05 GMT

Increase the heap size... refer to the blog
http://kindleconsulting.com/blog/entry/managing-jvm-heap-size-in-talend-open-studio.html
thanks
Vaibhav

Re: [resolved] OutOfMemoryError: GC overhead limit exceeded in simple MSSqBulkExec jb

Anonymous — Fri, 22 Aug 2014 18:27:22 GMT

Hi sanvaibhav and holberger,
increasing the heap space is mostly not a good idea. As kzone already mentioned it helps only for a short while and is never a solution. You have to redesign your job to process less data at once and take care you do not read to much data into the memory.
Figure out in which way you can separate the amount of records to a number of bunch of records and iterate over these bunches.

Re: [resolved] OutOfMemoryError: GC overhead limit exceeded in simple MSSqBulkExec jb

soujanyam — Sat, 23 Aug 2014 05:12:13 GMT

Thanks for your reply,
I split the input directory into parts and read them with different tFilelist and appending them through tUnite. For two directories(less than 50%data) getting result. But for total data not getting.
What might be the proper solution for this?

Re: [resolved] OutOfMemoryError: GC overhead limit exceeded in simple MSSqBulkExec jb

Anonymous — Sat, 23 Aug 2014 06:34:54 GMT

Your use case is to iterate through a very large list of files? Unfortunately the tFileList holds a representation of every file (the java.io.File object) in the memory and this causes sometimes a OutOfMemoryExceprion. The only way - currently - to solve this is to filter the files to prevent reading all of them.

Re: [resolved] OutOfMemoryError: GC overhead limit exceeded in simple MSSqBulkExec jb

soujanyam — Sat, 23 Aug 2014 06:50:05 GMT

Thanks for your prompt reply,
Thats why i used two tFilelists joins with tUnite follows with tUnite again,another join to this is from another tFilelist and etc.
Even though not getting.
What could I do?

Re: [resolved] OutOfMemoryError: GC overhead limit exceeded in simple MSSqBulkExec jb

soujanyam — Sat, 23 Aug 2014 08:01:58 GMT

My requirement is to read the all files from directory in single job and process further.Thats why I used multiple tFilelists with sub directories as input and to join them using tUnite.
Is there any other way to resolve my problem.
Thanks in advance.

Re: [resolved] OutOfMemoryError: GC overhead limit exceeded in simple MSSqBulkExec jb

Anonymous — Sat, 23 Aug 2014 12:25:00 GMT

The “java.lang.OutOfMemoryError: GC overhead limit exceeded” error indicates that GC has been trying to free the memory but is pretty much unable to get any job done. By default it happens when the JVM is spending more than 98% of the total time in GC and after GC less than 2% of the heap is recovered.

What would happen if this GC overhead limit was not present? Note that the “ GC overhead limit exceeded” error is thrown only when 2% of the memory was freed after several GC cycles. This means that the little amount GC was able to clean will be quickly filled again thus forcing GC to restart the cleaning process again.This forms a vicious cycle where the CPU is 100% busy with GC and no actual work can be done. End users of the application are facing extreme slowdowns – operations which used to be completed in milliseconds are now likely to take minutes to finish.
So the “java.lang.OutOfMemoryError: GC overhead limit exceeded” message is a pretty nice example of a fail fast principle in action. Bearing this in mind, disabling this check via -XX:-UseGCOverheadLimit is possibly the worst thing to do. You are just masking the symptoms instead of solving the problem.

A fast way way to give (temporary) relief to GC is to give more memory to the process. Again this is as easy as adding (or increasing if present) just one parameter in your startup scripts (-Xmx) similar to the following example:

java -Xmx1024m com.yourcompany.YourClass
If you wish to understand the underlying cause in more details, see examples and more advanced solutions, see the detailed post about java.lang.OutOfMemoryError: GC overhead limit exceeded error from https://plumbr.eu/outofmemoryerror/gc-overhead-limit-exceeded

Re: [resolved] OutOfMemoryError: GC overhead limit exceeded in simple MSSqBulkExec jb

soujanyam — Sat, 23 Aug 2014 13:39:17 GMT

Thanks for your reply. I didn't understand below one
java -Xmx1024m com.yourcompany.YourClass

Where I should add this in .ini fciprt or anything else and what is yourcopmpany and yourclass
Please reply me

Re: [resolved] OutOfMemoryError: GC overhead limit exceeded in simple MSSqBulkExec jb

Anonymous — Sat, 23 Aug 2014 19:19:06 GMT

You misunderstood me. It is not a solution to use two tFileList components because all the magic happens in the same job run -> same VM -> same memory. You have to design your job in a way it takes less work, process it and finish and need to be restarted to continue with the next portion of files or what ever you have to process.

Re: [resolved] OutOfMemoryError: GC overhead limit exceeded in simple MSSqBulkExec jb

soujanyam — Mon, 25 Aug 2014 07:04:47 GMT

Sorry sir, I could not able to understand.
You have to design your job in a way it takes less work, process it and finish and need to be restarted to continue with the next portion of files or what ever you have to process.
As I said, my requirement is to read the all files from directory in single job and process further.Thats why I used multiple tFilelists with sub directories as input and to join them using tUnite.

Re: [resolved] OutOfMemoryError: GC overhead limit exceeded in simple MSSqBulkExec jb

Anonymous — Mon, 25 Aug 2014 09:19:11 GMT

my requirement is to read the all files from directory in single job and process further

but it seems that you cannot ... because of allocated memory at JVM for your job.
Using an image, with your car, you cannot do more miles even if you put gazoline in 2 reservoir (your tFileLIst)
you can increase Allocated jvm memory to give enought space(heap) for your job ? Could be a solution
As jLolling as already said, process it in several time ... it's the same job (your requirement) but you NOT process ALL the file in the same time but in several times to release the memory between each process.
regards
laurent

Re: [resolved] OutOfMemoryError: GC overhead limit exceeded in simple MSSqBulkExec jb

soujanyam — Mon, 25 Aug 2014 09:44:29 GMT

Thanks for your reply laurent I tried like that with only one tFilelist once and then iterate the output with remaining input. but no use