Clean data in memory when a job ends

[_AnonymousUser]

I'm heaving memory problems when I run some jobs from Eclipse (on JBoss server).
When a job ends, the data that was being saved in memory is deleted, and is the memory released?
The error I'm getting is:
2014-02-25 09:51:14,780 ERROR (Thread-2 (group:HornetQ-client-global-threads-255292421)) javax.ejb.EJBTransactionRolledbackException: Unexpected Error
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:640)
at test_di.job_0_1.job.runJobInTOS(job.java:4586)
at test_di.job_0_1.job.runJob(job.java:4350)
at test_di.job_0_1.job.tRunJob_2Process(job.java:674)
at test_di.job_0_1.job.tRunJob_1Process(job.java:533)
at test_di.job_0_1.job.runJobInTOS(job.java:1050)
at test_di.job_0_1.job.runJob(job.java:914)
at pt.ptinovacao.agorang.qoscollector.RunJob.execute(RunJob.java:34)
at pt.ptinovacao.agorang.qoscollector.QueueListenerMDB.onMessage(QueueListenerMDB.java:122)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.jboss.aop.joinpoint.MethodInvocation.invokeTarget(MethodInvocation.java:122)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111)
at org.jboss.ejb3.EJBContainerInvocationWrapper.invokeNext(EJBContainerInvocationWrapper.java:69)
at org.jboss.ejb3.interceptors.aop.InterceptorSequencer.invoke(InterceptorSequencer.java:76)
at org.jboss.ejb3.interceptors.aop.InterceptorSequencer.aroundInvoke(InterceptorSequencer.java:62)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.jboss.aop.advice.PerJoinpointAdvice.invoke(PerJoinpointAdvice.java:174)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor.fillMethod(InvocationContextInterceptor.java:72)
at org.jboss.aop.advice.org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor_z_fillMethod_1575744261.invoke(InvocationContextInterceptor_z_fillMethod_1575744261.java)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor.setup(InvocationContextInterceptor.java:88)
at org.jboss.aop.advice.org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor_z_setup_1575744261.invoke(InvocationContextInterceptor_z_setup_1575744261.java)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:62)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:56)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.tx.NullInterceptor.invoke(NullInterceptor.java:42)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:68)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.aspects.tx.TxPolicy.invokeInCallerTx(TxPolicy.java:126)
at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:194)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.tx.NullInterceptor.invoke(NullInterceptor.java:42)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.security.Ejb3AuthenticationInterceptorv2.invoke(Ejb3AuthenticationInterceptorv2.java:80)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.BlockContainerShutdownInterceptor.invoke(BlockContainerShutdownInterceptor.java:67)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.aspects.currentinvocation.CurrentInvocationInterceptor.invoke(CurrentInvocationInterceptor.java:67)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.interceptor.EJB3TCCLInterceptor.invoke(EJB3TCCLInterceptor.java:86)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.mdb.MessagingContainer.localInvoke(MessagingContainer.java:282)
at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.delivery(MessageInflowLocalProxy.java:299)
at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.invoke(MessageInflowLocalProxy.java:152)
at $Proxy313.onMessage(Unknown Source)
at org.hornetq.ra.inflow.HornetQMessageHandler.onMessage(HornetQMessageHandler.java:256)
at org.hornetq.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:822)
at org.hornetq.core.client.impl.ClientConsumerImpl.access$100(ClientConsumerImpl.java:46)
at org.hornetq.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:940)
at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

[_AnonymousUser]

By the way, I'm using the tMap component, followed by the tFlowToIterate.
If I 'Store on Disk' the data in the tMap and I reduce the size of the Buffer (nb of rows), may it help me?

[Anonymous]

Hi,
The memory should be automatically recycled in java code.
For a large set of data, try to store the data on disk instead of memory on tMap. Also, allocate more memory to execute the job if you have 32G memory.
Please see KB article TalendHelpCenter:ExceptionOutOfMemory.
Best regards
Sabrina

[_AnonymousUser]

Thanks.
But tell me, how does the option 'Store data on disk' work? I never see any data being saved on the folder that I set as the target folder. And what value should be written in the Buffer size?

[Anonymous]

Assuming there is a join on tMap, the lookup data are stored in a temporary file and the file will be deleted automatically at the end of job, that's why you don't see the file in the specified directory. If you still have the OutOfMemory error, please upload a screenshot of your job, so that we can see which components you are using and give you some suggestions to optimize the job design.
Shong

[Anonymous]

Ok, thanks. I don't have Joins in my tMap component.
I attached my jobs (there are 2 types - one more complex than the other).

[Anonymous]

This error isn't necessarily related to the heap size, so caching to disk may not resolve the issue (and, in fact, may exacerbate it!). This can be indicative of the thread stack running out of capacity, if this is the problem it can be resolved with the -Xss option to the JVM. Try starting the process with a larger -Xss than it has now. If this option isn't being passed to the JVM, then this can also be due to the process running up against operating system limitations on files/processes for a user. If you're on linux you can test this with the ulimit command - the default for user processes is 1024 and can be increased with, say, ulimit -u 4096.
Hope this helps!