We are using Talend Big data Enterprise edition.
In our jobs, volume of data subject to change. Sometime, we will process 1 million and other time 10 million. Job is getting failed due default JVM memory settings.
I want to change JVM memory settings in runtime, depending on the amount of data we process. Triggering the job through unix. How to handle this case while running through shell? Kindle guide me.
Thanks
Hi,
So far, you can not change JVM setting when build a job in studio.
You can e
xport your jobs as Standalone Jobs(.sh or .bat) and change JVM setting from commandline.
Best regards
Sabrina
Thanks Sabrina.
Right now, our .sh has default settings as
java -Xms256M -Xmx1024M
But this need to be changed in runtime, while running the job depending on the amount of data, we are processing.
Thanks
You cannot change this setting while the job is running! It is not a lack of ability of Talend, it is simply the fact, these settings are initial settings for the JVM right at the start.
It is not a problem to allow much more memory with this parameters. The memory will only be allocated if it is needed.
Thanks jlolling.
If i run with default settings in .sh file, job fails due to out of memory issue for huge dataset.
java -Xms256M -Xmx1024M
You want us to allocate more memory(maximum anticipated depends on data in future)?. Any other way to handle this? Kindly help me.
Hi,
Have you tried to allocate more memory to your job to deal with your
out of memory issue before running?
You can open the Job which you want to allocate more memory to and in the Run view, open the Advanced Settings tab and select the
Use specific JVM arguments box. Then allocate more memory to the active Job by double clicking the default JVM arguments and editing them.
Best regards
Sabrina
