TAC becomes unreachable and unresponsive, so no Jobs can be run

Talend Version          6.x, 7.x
Key words TAC performance unresponsive unreachable
Product Talend Data Integration
Component Talend Administration Center
Article Type Debugging
Problem Description Talend Administration Center (TAC) needs to be restarted frequently as it gets blocked after some time.
Problem root cause

There could be many causes for this performance issue. It is important to know:

  • How many tasks are running at the same time?
  • How many tasks are running in a day on average?
  • For how long are they running?
  • When the tasks were created, was the "Generate on change" option checked?
  • How often are the related Jobs modified?

Collect TAC technical log, do a thread dump at the moment of the performance issue, collect TAC configuration files (configuration.properties and quartz.properties)

 

- If using MySQL, changing the transaction isolation level can help.

 

- If using Microsoft SQL Server 2008, read-committed isolation using row versioning must be activated.

 

- TAC's Tomcat memory configuration parameters should also be reviewed: setenv.bat (Windows) or the setenv.sh (Linux) :

- Xms,  -Xmx, -XX:MaxMetaspaceSize

It is found in the path: 

C:\Talend\7.x.x\tac\apache-tomcat\bin

OR

<TomcatPath>/bin

 

to  tune TAC memory settings :  https://community.talend.com/t5/Migration-Configuration-and/How-do-I-modify-TAC-or-MDM-memory-settin...

 

- In addition restarting the Apache-Tomcat and clearing the apache-tomcat/work and apache-tomcat/temp directories post parameter updates is a necessary step. Lastly upping the ulimit from the OS default of 1024 to an excess of 10k is also a good idea. 

 

ulimit -n 10000

Solution or Workaround

Per the online document (Recommendations about environment and configuration for the Talend Administration Center) you should look at the values of these three parameters:

  • hibernate.c3p0.max_size (set in the configuration.properties file)
  • org.quartz.threadPool.threadCount (set in the quartz.properties file)
  • org.quartz.dataSource.QRTZ_DS.maxConnections (set in the quartz.properties file)

Note: You can locate the configuration.properties and the quartz.properties files in the Tomcat8.0\webapps\org.talend.administrator-7.2.1\WEB-INF\classes folder.

 

Increasing these values can be done incrementally, and it depends on the value of MAX_CONCURRENT_TASK_EXECUTIONS + MAX_CONCURRENT_PLAN_EXECUTIONS.

 

If hibernate.c3p0.max_size < org.quartz.dataSource.QRTZ_DS.maxConnections you will have problems, so it is important to ensure that the hibernate.c3p0.max_size > org.quartz.dataSource.QRTZ_DS.maxConnections.

 

You will need to restart the Tomcat / TAC for the change to take effect. After changing the values, the TAC should behave well and stop crashing.

Default values

(small install)

# quartz.properties

org.quartz.threadPool.threadCount = 30

org.quartz.dataSource.QRTZ_DS.maxConnections = 30

 

# configuration.properties

hibernate.c3p0.max_size=32

 

Example values 1

(medium install)

# quartz.properties

org.quartz.threadPool.threadCount = 60

org.quartz.dataSource.QRTZ_DS.maxConnections = 70

 

# configuration.properties

hibernate.c3p0.max_size=80

 

Example values 2

(larger install)

# quartz.properties

org.quartz.threadPool.threadCount = 100

org.quartz.dataSource.QRTZ_DS.maxConnections = 110

 

# configuration.properties

hibernate.c3p0.max_size=120

 

Example values 3

(enterprise install)

# quartz.properties

org.quartz.threadPool.threadCount = 400

org.quartz.dataSource.QRTZ_DS.maxConnections = 430

 

# configuration.properties

hibernate.c3p0.max_size=460

 

Version history
Revision #:
17 of 17
Last update:
‎02-07-2020 05:44 AM
Updated by: