Memory leak

[Anonymous]

Hello there,
I don't know if any of you encountered this but the memory usage of my jobs is increasing over time to the point where it uses more than the system allows and eventually erroring because the system is unable to allocate more memory. I believe it might be from the job setup i currently have but i would like your opinion. Here is a short description of the platform and the jobs (detailed screenshots of the jobs attached).

All the jobs are running on 4Gb RAM unix machines. 
There are 4 jobs per machine.
The memory usage increase is about 50Mb over 4h. 
All the jobs are started using the shell script with the following memory parameters:  -Xms256M -Xmx1024M
Each scripts starts a "container" job that does the following:

tMomConnection --- tPosgresqlConection --- tMomInput --- tRunJob --- tMomOutput --- tMomCommit

The job that they run (tRunJob) is as follows:

tExtractJsonField --- tPostgresqlOutput --- tPostgresqlCommit --- tBufferOutput

Is there any reason known to you why this kind of job might have a growing memory consumption ? I have tried to find an answer by myself but i'm really stuck there, a bit of help would be so appreciated. 

[Anonymous]

1-You are putting all the data in memory before doing the pivot. your data may exceed 4GB Ram
2-if you are probably catching a lot of errors , these are stored in memery before being sent to the parent job. why wouldn't you write the logs to a table or a file ?
as an alternative you can do an iterate flow from the parent to the child job , in that case you control the mem but your job will be a bit slower
Or
Add more memory add more -Xms in your job params

[Anonymous]

Hi ahallam,
Thank you for your answer, 

I don't see quite what you mean when you say i put all the data in memory before the pivot ? Would this be a problem if the data is rarely bigger than a mb ? What would be another solution ?
Every error is sent to the parent so it can send an email if the error is important and logs is otherwise. During the memory leak test i did no errors were reported so i guess no errors were caught. With the setup you see in the screenshots do you think errors could go unnoticed and generate a leak ? I will have do some tests to find out weather the errors can  
Adding more memory will not fix the fact that the memory consumption keeps growing it will eventually grow bigger than the available amount. This seems to be the standard answer for any talend performance issue, but performance is not the problem in my case it is the fact that the memory consumption never stops growing. 

[_AnonymousUser]

Hi All,
I am facing one issue where I am not able to commit the messages from the MQ after I have received the messages from the queue.Below is the sequence of components that I am using .
tMomInput  -->main row --> tlogRow --> On Component Ok --> tMomCommit
Everytime I run the job , can see the same message again , even if I send any message ..then that message is also being received alongwith old messages

Regards,
Harish

[Anonymous]

In TOS 7.0 Big Data, I noticed that Eclipse in Code view was warning on tLogRow (table formatted output, use log4j) about a possible resource leak.

java.util.Formatter formatter = new java.util.Formatter(new StringBuilder());  

resource leak: formatter is never closed.

It only complains in Mode=Table.  Basic and Vertical seem to be ok.

[Anonymous]

Hello,

Could you please give us some trouble screenshots about your issue?

Best regards

Sabrina

[Anonymous]

If  I'm reading this right, with every loop past this code, a new string formatter is being allocated. There are several posts in the forums warning about resource leaks in Formatter.

java.util.Formatter

Possible solutions:

https://www.java-forums.org/new-java/82232-fomatter-print.html

https://stackoverflow.com/questions/37748357/how-to-close-java-formatter-in-finally-or-not

Screenshot from 2018-06-11 15-42-26.png