topic Talend Memory Management in Talend Studio

Talend Memory Management

Anonymous — Thu, 09 May 2013 11:28:43 GMT

Hi Friends
We are using Talend to import 4.5 millions of records from CSV to MYSQL DB and processing the records. We need the following things to be done,
1. To clean the memory once the talend exe file runs successfully. Currently all the process are in the memory. I thought Talend will do garbage collection. Is there are way to clean the memory/kill tasks in Talend itself.
2. Customer want the talend exe to run in low/medium priority. Can I set it in talend.
Thanks For your valuable inputs and help.
Kumar

Re: Talend Memory Management

Anonymous — Thu, 09 May 2013 13:17:48 GMT

Im not sure your questions make alot of sense could you describe the problem you are trying to solve?
1) an exported Talend job will exit after completing successfully-- this will free any memory it was using back to the OS. The talend client will cache items in memory (like a good java program should) and so you will see it gradually use more and more untill it reaches the configured heap size. This is normal and should be fine.

2) Im not sure why you would want to do this, but you can change the priority on windows by starting Talend with the following switches: "%comspec% /c start /low <talend executable>"

Re: Talend Memory Management

Anonymous — Fri, 10 May 2013 06:54:05 GMT

Hi John
Thanks for your valuable inputs,
1. I wish it worked like that. I took a screenshot of Task Manager after I run the 4.5 million records 4 times (in regular interval) and I was able to see multiple processes created (e.g. cmd.exe), which was not cleaned by GC. Is there a way to invoke GC at the end of the package?. Also for components like tCSV Import does not have a dispose() method invoked or there is an option to check/uncheck dispose(). Screenshots attached.
2. I exported my talend job and scheduled it in windows scheduler(.exe). Can I set priority in talend when it is running in production machine.
Thanks for your time.

Re: Talend Memory Management

Anonymous — Fri, 10 May 2013 16:21:58 GMT

looks like your screenshots didnt get attached correctly, I think they may help us understand your problem a little better.

when you run your job on windows it should run under java.exe, not cmd.exe. If you are using cmd.exe to invoke the job, you may see one running-- but it should simply be a shell and take little memory. If you are seeing cmd.exe using large amounts of memory, I would suspect that process is not related to your jobs.

Re: Talend Memory Management

Anonymous — Fri, 10 May 2013 22:32:50 GMT

To your second question. It is not recommended to set the priority of a program under any OS and independently of Java or Talend. I have never seen any good influence to the performance but many problems for stability.
The Garbage Collector works within the Java Virtual Maschine and if a job ends, the GC has nothing to do because the JVM is going to shutdown. The GC does not clean processes from the OS!
dispose is not a common method for resource consuming component. Of course the tFileInputDelimited component closes the input stream but does not allocate very much memory because it does not need this.
What could happend is, if there is a huge amount of short living java objects (you read a very large and wide file) this could lead to some pause because the GC has to erase them.

Re: Talend Memory Management

Anonymous — Mon, 13 May 2013 06:45:57 GMT

Hi John
Thanks. Attaching the image again. Not sure why it was not attached last time. I selected '4' Image upload slots in 'Post A Reply' page from chrome.
Hi Jlolling
Thanks. Please help,
a. How to set priority in Java or Talend if setting priority in OS is not recommended.
b. If I understood your clarification correctly, you are saying, THE GC WILL CLEAN-UP ALL THE OBJECTS BY DEFAULT AS IT IS A JAVA DEFAULT FEATURE. SINCE THERE ARE MANY OBJECTS CREATED IT IS TAKING TIME. Right
Thanks and Kind Regards
Kumar