TAC becomes unreachable and unresponsive, so no Jobs can be run

Talend Version          5.x, 6.x
Key words TAC performance unresponsive unreachable
Product Talend Data Integration
Component Talend Administration Center
Article Type Debugging
Problem Description Talend Administration Center (TAC) needs to be restarted frequently as it gets blocked after 1 hour.
Problem root cause

There could be many causes for this performance issue. It is important to know:

  • How many tasks are running at the same time?
  • How many tasks are running in a day on average?
  • For how long are they running?
  • When the tasks were created, was the "Generate on change" option checked?
  • How often are the related Jobs modified?

To get the TAC technical log, do a thread dump at the moment of the performance issue.

 

If using MySQL, changing the transaction isolation level can help.

 

If using Microsoft SQL Server 2008, read-committed isolation using row versioning must be activated.

Solution or Workaround

Per the online document (Recommendations about environment and configuration for the Talend Administration Center) you should look at the values of these three parameters:

  • hibernate.c3p0.max_size (set in the configuration.properties file)
  • org.quartz.threadPool.threadCount (set in the quartz.properties file)
  • org.quartz.dataSource.QRTZ_DS.maxConnections (set in the quartz.properties file)

Note: You can locate the configuration.properties and the quartz.properties files in the Tomcat8.0\webapps\org.talend.administrator-6.2.1\WEB-INF\classes folder.

 

Increasing these values can be done incrementally, and it depends on the value of MAX_CONCURRENT_TASK_EXECUTIONS + MAX_CONCURRENT_PLAN_EXECUTIONS.

 

If hibernate.c3p0.max_size < org.quartz.dataSource.QRTZ_DS.maxConnections you will have problems, so it is important to ensure that the hibernate.c3p0.max_size > org.quartz.dataSource.QRTZ_DS.maxConnections.

 

You will need to restart the Tomcat / TAC for the change to take effect. After changing the values, the TAC should behave well and stop crashing.

Default values

(small install)

# quartz.properties

org.quartz.threadPool.threadCount = 30

org.quartz.dataSource.QRTZ_DS.maxConnections = 30

 

# configuration.properties

hibernate.c3p0.max_size=32

 

Example values 1

(medium install)

# quartz.properties

org.quartz.threadPool.threadCount = 60

org.quartz.dataSource.QRTZ_DS.maxConnections = 70

 

# configuration.properties

hibernate.c3p0.max_size=80

 

Example values 2

(larger install)

# quartz.properties

org.quartz.threadPool.threadCount = 100

org.quartz.dataSource.QRTZ_DS.maxConnections = 110

 

# configuration.properties

hibernate.c3p0.max_size=120

 

Version history
Revision #:
11 of 11
Last update:
‎09-29-2018 12:13 AM
Updated by:
 
Labels (2)
Comments
Employee

Another good thing to check related to the memory settings is the setenv.bat (Windows) or the setenv.sh (Linux) to see if the relevant parameters are set correctly for their environment. If they are small or the default values, increase these. The parameters are below:

 

-Xms

-Xmx

-XX:MaxMetaspaceSize

 

It is found in the path: 

 

C:\Talend\6.x.x\tac\apache-tomcat\bin

OR

<TomcatPath>/bin

 

Documentation:

https://help.talend.com/reader/vuI_X~V6unFjTgNxRMPcLw/Ij~7tBlW8im63rAGnGHT3A

 

 

Employee

In addition restarting the Apache-Tomcat and clearing the apache-tomcat/work and apache-tomcat/temp directories post parameter updates is a necessary step. Lastly upping the ulimit from the OS default of 1024 to an excess of 10k is a good idea here too. 

 

ulimit -n 10000

Employee

The default maximum amount of memory allotted for TAC is only 2GB -- we should increase that to at least 4GB, ideally 8GB or more.


To increase or adjust TAC memory settings:

 

https://community.talend.com/t5/Migration-Configuration-and/How-do-I-modify-TAC-or-MDM-memory-settin...

 

Employee

This is example of an 'Enterprise' install:

 

Example values 3

(enterprise install)

# quartz.properties

org.quartz.threadPool.threadCount = 400

org.quartz.dataSource.QRTZ_DS.maxConnections = 430

 

# configuration.properties

hibernate.c3p0.max_size=460