Clean data in memory when a job ends

_AnonymousUser — Tue, 25 Feb 2014 11:14:24 GMT

I'm heaving memory problems when I run some jobs from Eclipse (on JBoss server).
When a job ends, the data that was being saved in memory is deleted, and is the memory released?
The error I'm getting is:
2014-02-25 09:51:14,780 ERROR (Thread-2 (group:HornetQ-client-global-threads-255292421)) javax.ejb.EJBTransactionRolledbackException: Unexpected Error
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:640)
at test_di.job_0_1.job.runJobInTOS(job.java:4586)
at test_di.job_0_1.job.runJob(job.java:4350)
at test_di.job_0_1.job.tRunJob_2Process(job.java:674)
at test_di.job_0_1.job.tRunJob_1Process(job.java:533)
at test_di.job_0_1.job.runJobInTOS(job.java:1050)
at test_di.job_0_1.job.runJob(job.java:914)
at pt.ptinovacao.agorang.qoscollector.RunJob.execute(RunJob.java:34)
at pt.ptinovacao.agorang.qoscollector.QueueListenerMDB.onMessage(QueueListenerMDB.java:122)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.jboss.aop.joinpoint.MethodInvocation.invokeTarget(MethodInvocation.java:122)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111)
at org.jboss.ejb3.EJBContainerInvocationWrapper.invokeNext(EJBContainerInvocationWrapper.java:69)
at org.jboss.ejb3.interceptors.aop.InterceptorSequencer.invoke(InterceptorSequencer.java:76)
at org.jboss.ejb3.interceptors.aop.InterceptorSequencer.aroundInvoke(InterceptorSequencer.java:62)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.jboss.aop.advice.PerJoinpointAdvice.invoke(PerJoinpointAdvice.java:174)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor.fillMethod(InvocationContextInterceptor.java:72)
at org.jboss.aop.advice.org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor_z_fillMethod_1575744261.invoke(InvocationContextInterceptor_z_fillMethod_1575744261.java)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor.setup(InvocationContextInterceptor.java:88)
at org.jboss.aop.advice.org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor_z_setup_1575744261.invoke(InvocationContextInterceptor_z_setup_1575744261.java)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:62)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:56)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.tx.NullInterceptor.invoke(NullInterceptor.java:42)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:68)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.aspects.tx.TxPolicy.invokeInCallerTx(TxPolicy.java:126)
at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:194)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.tx.NullInterceptor.invoke(NullInterceptor.java:42)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.security.Ejb3AuthenticationInterceptorv2.invoke(Ejb3AuthenticationInterceptorv2.java:80)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.BlockContainerShutdownInterceptor.invoke(BlockContainerShutdownInterceptor.java:67)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.aspects.currentinvocation.CurrentInvocationInterceptor.invoke(CurrentInvocationInterceptor.java:67)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.interceptor.EJB3TCCLInterceptor.invoke(EJB3TCCLInterceptor.java:86)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.mdb.MessagingContainer.localInvoke(MessagingContainer.java:282)
at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.delivery(MessageInflowLocalProxy.java:299)
at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.invoke(MessageInflowLocalProxy.java:152)
at $Proxy313.onMessage(Unknown Source)
at org.hornetq.ra.inflow.HornetQMessageHandler.onMessage(HornetQMessageHandler.java:256)
at org.hornetq.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:822)
at org.hornetq.core.client.impl.ClientConsumerImpl.access$100(ClientConsumerImpl.java:46)
at org.hornetq.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:940)
at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

Re: Clean data in memory when a job ends

_AnonymousUser — Tue, 25 Feb 2014 12:31:21 GMT

By the way, I'm using the tMap component, followed by the tFlowToIterate.
If I 'Store on Disk' the data in the tMap and I reduce the size of the Buffer (nb of rows), may it help me?

Re: Clean data in memory when a job ends

Anonymous — Wed, 26 Feb 2014 08:46:31 GMT

Hi,
The memory should be automatically recycled in java code.
For a large set of data, try to store the data on disk instead of memory on tMap. Also, allocate more memory to execute the job if you have 32G memory.
Please see KB article TalendHelpCenter:ExceptionOutOfMemory.
Best regards
Sabrina

Re: Clean data in memory when a job ends

_AnonymousUser — Thu, 27 Feb 2014 16:13:40 GMT

Thanks.
But tell me, how does the option 'Store data on disk' work? I never see any data being saved on the folder that I set as the target folder. And what value should be written in the Buffer size?

Re: Clean data in memory when a job ends

Anonymous — Fri, 28 Feb 2014 02:14:55 GMT

Assuming there is a join on tMap, the lookup data are stored in a temporary file and the file will be deleted automatically at the end of job, that's why you don't see the file in the specified directory. If you still have the OutOfMemory error, please upload a screenshot of your job, so that we can see which components you are using and give you some suggestions to optimize the job design.
Shong

Re: Clean data in memory when a job ends

Anonymous — Fri, 28 Feb 2014 09:43:13 GMT

Ok, thanks. I don't have Joins in my tMap component.
I attached my jobs (there are 2 types - one more complex than the other).

Re: Clean data in memory when a job ends

Anonymous — Fri, 28 Feb 2014 17:03:18 GMT

This error isn't necessarily related to the heap size, so caching to disk may not resolve the issue (and, in fact, may exacerbate it!). This can be indicative of the thread stack running out of capacity, if this is the problem it can be resolved with the -Xss option to the JVM. Try starting the process with a larger -Xss than it has now. If this option isn't being passed to the JVM, then this can also be due to the process running up against operating system limitations on files/processes for a user. If you're on linux you can test this with the ulimit command - the default for user processes is 1024 and can be increased with, say, ulimit -u 4096.
Hope this helps!

topic Clean data in memory when a job ends in Talend Studio

Clean data in memory when a job ends

Re: Clean data in memory when a job ends

Re: Clean data in memory when a job ends

Re: Clean data in memory when a job ends

Re: Clean data in memory when a job ends

Re: Clean data in memory when a job ends

Re: Clean data in memory when a job ends