tFileInputJSON gives "GC Overhead Limit" error

[Anonymous]

Hello Champ,

I have created a simple job:

tFileInputJSON --> (main) --> tLogRow

The source JSON file is around 2GB size. I even increased JVM in RUN to Xms2G, Xmx4G. But always fails with memory issue.

NOTE: the JSON works great if it is a simple file.

Is there a way to extract big size file? or is it a product limitation? I seen some articles for CSV file, nothing for JSON

Looking forward to hear some valuable answers. Thanks.

regards,

K

[manodwhb]

@rvkiruba,what is the system RAM size? and if you are using enterprise edition you can execute on remote system.

[Anonymous]

@manodwhb my PC has 8GB RAM. Is it not sufficient?

Win 7 Enterprise. Can you pls brief me about remote system?

Thanks.

K

[Anonymous]

Hello,

Could you please let us know if this article helps?

https://community.talend.com/t5/Migration-Configuration-and/GC-overhead-limit-error-when-running-Job...

Best regards

Sabrina

[Anonymous]

Hi Sabrina,

Thanks for your reply. The article is not helping me. I get below error now.

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space

I too followed some other instructions and updated JVM like below, but no luck. 

With 8 GB of memory available on 64-bit system, the optimal settings can be:
-Xms1024m
-Xmx4096m
-XX:MaxPermSize=512m
-Dfile.encoding=UTF-8

Thanks.

K

[Anonymous]

Hello,

Could you please try to open the Job to which you want to allocate more memory and in the Run view, open the Advanced Settings tab and select the Use specific JVM arguments box. Please allocate more memory to the active Job by double-clicking the default JVM arguments and editing them.

This change only applies for the active Job. The JVM settings will persist in the job script, and take effect when the job is exported and is executed outside Talend Studio.

Let us know if it is OK with you.

Best regards

Sabrina

[Anonymous]

Hi Sabrina,

As i said in my previous post, i have made necessary changes to JVM in Run --> Advanced Settings and ran the job in my PC. Screenshot attached fyr.

Are you saying that, the JVM setting change work only when the job runs in cloud. Is that correct?

Thanks,

K

talend_jvm_settings.JPG

[Anonymous]

Hello,

It means the change only applies for your current Job not for the whole studio. And the JVM settings will persist in the job script, and take effect when your job is exported and is executed outside Talend Studio(.bat or .sh file).

Best regards

Sabrina

[Anonymous · Accepted solution]

Thanks Sabrina. It didn't work at all. We have found alternative solution to split JSON file into multiple small files and processed successfully.

[Anonymous]

Hello,

Thanks for sharing your solution with us on forum.

Best regards

Sabrina