TAC becomes too slow with 40+ tasks running

Problem Description

TAC becomes unusable when running 40 tasks at the same time, with the TAC > Configuration and TAC > Job Conductor pages refreshing continuously.

 

If you run these 40 tasks manually (choosing all 40 tasks and clicking Run), some tasks display the status of Running, some display Request Running, and others remain in the Ready to Run state.

 

This may lead to the database connection closing with the following exception:

 

org.talend.administrator.common.exception.SystemException: org.talend.administrator.common.exception.DBException: !!!Cannot create session or transaction.!!!
	at org.talend.administrator.common.config.persistence.ConfigHelper.findValueByKey(ConfigHelper.java:91)
	at org.talend.administrator.common.config.persistence.ConfigHelper.findValueByKey(ConfigHelper.java:62)
	at org.talend.administrator.common.config.ConfigHandler.getValue(ConfigHandler.java:784)
	at org.talend.administrator.common.config.ConfigHandler.getBooleanValue(ConfigHandler.java:979)
	at org.talend.administrator.common.persistence.dbconfiguration.PropertiesDatabaseConfiguration.isDoDatabaseEmbeddedBackupAtShutdown(PropertiesDatabaseConfiguration.java:251)
	at org.talend.administrator.common.persistence.servlet.EmbeddedDbCloser.contextDestroyed(EmbeddedDbCloser.java:42)
	at org.apache.catalina.core.StandardContext.listenerStop(StandardContext.java:4775)
	at org.apache.catalina.core.StandardContext.stopInternal(StandardContext.java:5385)
	at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:232)
	at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1425)
	at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1414)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.talend.administrator.common.exception.DBException: !!!Cannot create session or transaction.!!!
	at org.talend.administrator.common.persistence.hibernate.HibernatePersistenceAdapter.createTransaction(HibernatePersistenceAdapter.java:119)
	at org.talend.administrator.common.persistence.hibernate.HibernatePersistenceAdapter.retrieveUnique(HibernatePersistenceAdapter.java:1382)
	at org.talend.administrator.common.config.persistence.ConfigHelper.findValueByKey(ConfigHelper.java:82)
	... 14 more
Caused by: org.talend.administrator.common.exception.DBException: !!!Cannot commit transaction.!!!
	at org.talend.administrator.common.persistence.hibernate.HibernateTransactionHolder.beginTransaction(HibernateTransactionHolder.java:173)
	at org.talend.administrator.common.persistence.hibernate.HibernateTransactionHolder.<init>(HibernateTransactionHolder.java:47)
	at org.talend.administrator.common.persistence.hibernate.HibernatePersistenceAdapter.createTransaction(HibernatePersistenceAdapter.java:112)
	... 16 more
Caused by: org.hibernate.exception.GenericJDBCException: Cannot open connection
	at org.hibernate.exception.SQLStateConverter.handledNonSpecificException(SQLStateConverter.java:103)
	at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:91)
	at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:43)
	at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:29)
	at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:420)
	at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:144)
	at org.hibernate.jdbc.JDBCContext.connection(JDBCContext.java:119)
	at org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:57)
	at org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1326)
	at org.talend.administrator.common.persistence.hibernate.HibernateTransactionHolder.beginTransaction(HibernateTransactionHolder.java:167)
	... 18 more
Caused by: org.h2.jdbc.JdbcSQLException: Database is already closed (to disable automatic closing at VM shutdown, add ";DB_CLOSE_ON_EXIT=FALSE" to the db URL) [90121-187]
	at org.h2.message.DbException.getJdbcSQLException(DbException.java:345)
	at org.h2.message.DbException.get(DbException.java:179)
	at org.h2.message.DbException.get(DbException.java:155)
	at org.h2.message.DbException.get(DbException.java:144)
	at org.h2.jdbc.JdbcConnection.checkClosed(JdbcConnection.java:1469)
	at org.h2.jdbc.JdbcConnection.checkClosed(JdbcConnection.java:1444)
	at org.h2.jdbc.JdbcConnection.getAutoCommit(JdbcConnection.java:447)
	at com.mchange.v2.c3p0.impl.NewProxyConnection.getAutoCommit(NewProxyConnection.java:985)
	at org.hibernate.connection.C3P0ConnectionProvider.getConnection(C3P0ConnectionProvider.java:37)
	at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:417)
	... 23 more

 

Root Cause

According to the exception, this error happens when the DB was shutdown or the limit of connections was exceeded.

 

Solution

  1. Edit tomcat/webapps/org.talend.administrator-6.1.1/WEB-INF/classes/configuration.properties to change the default settings for TAC from 32 to 100.

    hibernate.c3p0.max_size=100
  2. Restart TAC.

  3. Run 50 tasks at same time, which should all run without exceptions.
Version history
Revision #:
6 of 6
Last update:
‎05-09-2018 07:19 PM
Updated by:
 
Contributors