Too many threads on Tomcat for Talend

[Anonymous]

Hi,

We have a Talend 5.2 MDM system. The TAC is deployed on a Tomcat 7.0.42 server. We are facing a issue in which there are threads being created by the TAC on tomcat and after some time, the threads hang and the jobs return a timeout exception.

We did a grep on the server for the talend process id and currently there are 190+ threads running. Any help would be appreciated.

talend   17581     1 17585  0  183 Sep14 ?        00:00:02 /usr/java/jdk1.7.0_25/bin/java -Djava.util.logging.config.file=/talend/tac/apache-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.awt.headless=true -Dfile.encoding=UTF-8 -server -Xms1024m -Xmx1024m -XX:NewSize=512m -XX:MaxNewSize=512m -XX:PermSize=512m -XX:MaxPermSize=512m -XX:SurvivorRatio=8 -XX:+UseParallelGC -XX:ParallelGCThreads=4 -XX:+UseLargePages -Djava.endorsed.dirs=/talend/tac/apache-tomcat/endorsed -classpath /talend/tac/apache-tomcat/bin/bootstrap.jar:/talend/tac/apache-tomcat/bin/tomcat-juli.jar -Dcatalina.base=/talend/tac/apache-tomcat -Dcatalina.home=/talend/tac/apache-tomcat -Djava.io.tmpdir=/talend/tac/apache-tomcat/temp org.apache.catalina.startup.Bootstrap start

 A restart clears the issue, but this happens intermittently. 

[Anonymous]

Hello,

Is there any more error message which is located in <ServersInstallationPath>\apache-tomcat-XXX\logs?

Best regards

Sabrina

[Anonymous]

Tomcat 7 , default allow 200 threads, looks like your tomcat reach to that level & then hung.

If you want to change the maxthread , you need to change in org.apache.catalina.Executor java file or you can change at catalina level by changing, server.xml as below:

<connector connectiontimeout="20000"
           maxthreads="400"
           port="8080"
           protocol="HTTP/1.1"
           redirectport="8443" />

[Anonymous]

Hello,

Is there any update for your issue?

Best regards

Sabrina

[Anonymous]

Hi, 

Sorry for the late update, the issue still remain. 

<Connector port="8080" protocol="HTTP/1.1"
connectionTimeout="300000" compression="force"
maxThreads="1000" maxHttpHeaderSize="8192" acceptCount="1000"
redirectPort="8443" minSpareThreads="100" maxSpareThreads="105" />

This is the server.xml in Tomcat, as you can see maxThreads is a high enough number. We also tried to tune up the tomcat java_opts using the following: 

JAVA_OPTS="$JAVA_OPTS -Dfile.encoding=UTF8 -Dorg.apache.catalina.loader.WebappClassLoader.ENABLE_CLEAR_REFERENCES=false -server -d64 -Xms2048m -Xmx2048m -XX:NewSize=512m -XX:MaxNewSize=512m -XXermSize=256m -XX:MaxPermSize=512m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+CMSCompactWhenClearAllSoftRefs -XX:CMSInitiatingOccupancyFraction=25 -XX:+CMSScavengeBeforeRemark -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XXarallelCMSThreads=2 -XX:+UseLargePages -XX:+UseCompressedOops -XX:+DisableExplicitGC -XX:-UseBiasedLocking -XX:+BindGCTaskThreadsToCPUs -XX:MaxTenuringThreshold=0 -XXurvivorRatio=25 -XXarallelGCThreads=8"

But the issue remains.

Thanks

[Anonymous]

Hi,

I found a stackOverflow link in which it says the thread being kept alive might be a mySQL connector jar problem.

https://stackoverflow.com/questions/13161747/where-do-i-have-to-place-the-jdbc-driver-for-tomcats-co...

The solution suggests to place the mysql connector jar in the Tomcat/lib folder and not in the web app's WEB-INF/lib folder. Now my question is, will placing the jar in the tomcat/lib folder and deleting the jar from all the web app's WEB-INF/lib folder be a problem?

Keep in mind that this is a production environment.

Thanks

[Anonymous]

Hi,

For your production environment, could you please create a case on talend support portal so that we can give you a remote assistance(webex) through support cycle with priority?

Best regards

Sabrina