Too many threads on Tomcat for Talend

Five Stars

Too many threads on Tomcat for Talend

Hi,

 

We have a Talend 5.2 MDM system. The TAC is deployed on a Tomcat 7.0.42 server. We are facing a issue in which there are threads being created by the TAC on tomcat and after some time, the threads hang and the jobs return a timeout exception.

 

We did a grep on the server for the talend process id and currently there are 190+ threads running. Any help would be appreciated.

 

talend   17581     1 17585  0  183 Sep14 ?        00:00:02 /usr/java/jdk1.7.0_25/bin/java -Djava.util.logging.config.file=/talend/tac/apache-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.awt.headless=true -Dfile.encoding=UTF-8 -server -Xms1024m -Xmx1024m -XX:NewSize=512m -XX:MaxNewSize=512m -XX:PermSize=512m -XX:MaxPermSize=512m -XX:SurvivorRatio=8 -XX:+UseParallelGC -XX:ParallelGCThreads=4 -XX:+UseLargePages -Djava.endorsed.dirs=/talend/tac/apache-tomcat/endorsed -classpath /talend/tac/apache-tomcat/bin/bootstrap.jar:/talend/tac/apache-tomcat/bin/tomcat-juli.jar -Dcatalina.base=/talend/tac/apache-tomcat -Dcatalina.home=/talend/tac/apache-tomcat -Djava.io.tmpdir=/talend/tac/apache-tomcat/temp org.apache.catalina.startup.Bootstrap start

 A restart clears the issue, but this happens intermittently. 

Moderator

Re: Too many threads on Tomcat for Talend

Hello,

Is there any more error message which is located in <ServersInstallationPath>\apache-tomcat-XXX\logs?

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Employee

Re: Too many threads on Tomcat for Talend

Tomcat 7 , default allow 200 threads, looks like your tomcat reach to that level & then hung.

If you want to change the maxthread , you need to change in org.apache.catalina.Executor java file or you can change at catalina level by changing, server.xml as below:

 

<connector connectiontimeout="20000"
           maxthreads="400"
           port="8080"
           protocol="HTTP/1.1"
           redirectport="8443" />
Moderator

Re: Too many threads on Tomcat for Talend

Hello,

Is there any update for your issue?

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Five Stars

Re: Too many threads on Tomcat for Talend

Hi, 

Sorry for the late update, the issue still remain. 

 

 

<Connector port="8080" protocol="HTTP/1.1"
connectionTimeout="300000" compression="force"
maxThreads="1000" maxHttpHeaderSize="8192" acceptCount="1000"
redirectPort="8443" minSpareThreads="100" maxSpareThreads="105" />

 

 

 

This is the server.xml in Tomcat, as you can see maxThreads is a high enough number. We also tried to tune up the tomcat java_opts using the following: 

 

JAVA_OPTS="$JAVA_OPTS -Dfile.encoding=UTF8 -Dorg.apache.catalina.loader.WebappClassLoader.ENABLE_CLEAR_REFERENCES=false -server -d64 -Xms2048m -Xmx2048m -XX:NewSize=512m -XX:MaxNewSize=512m -XXSmiley TongueermSize=256m -XX:MaxPermSize=512m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+CMSCompactWhenClearAllSoftRefs -XX:CMSInitiatingOccupancyFraction=25 -XX:+CMSScavengeBeforeRemark -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XXSmiley TonguearallelCMSThreads=2 -XX:+UseLargePages -XX:+UseCompressedOops -XX:+DisableExplicitGC -XX:-UseBiasedLocking -XX:+BindGCTaskThreadsToCPUs -XX:MaxTenuringThreshold=0 -XXSmiley FrustratedurvivorRatio=25 -XXSmiley TonguearallelGCThreads=8"

 

 

But the issue remains.

 

Thanks

Highlighted
Five Stars

Re: Too many threads on Tomcat for Talend

Hi,

 

I found a stackOverflow link in which it says the thread being kept alive might be a mySQL connector jar problem.

https://stackoverflow.com/questions/13161747/where-do-i-have-to-place-the-jdbc-driver-for-tomcats-co...

 

The solution suggests to place the mysql connector jar in the Tomcat/lib folder and not in the web app's WEB-INF/lib folder. Now my question is, will placing the jar in the tomcat/lib folder and deleting the jar from all the web app's WEB-INF/lib folder be a problem?

 

Keep in mind that this is a production environment.

Thanks

Moderator

Re: Too many threads on Tomcat for Talend

Hi,

For your production environment, could you please create a case on talend support portal so that we can give you a remote assistance(webex) through support cycle with priority?

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.

What’s New for Talend Spring ’19

Watch the recorded webinar!

Watch Now

Tutorial

Introduction to Talend Open Studio for Data Integration.

Watch

Downloads and Trials

Test drive Talend's enterprise products.

Downloads

Definitive Guide to Data Integration

Practical steps to developing your data integration strategy.

Download