One Star

Talend Jobs stop running for unknown reason

Hi, we are new to Talend. Recently we have deployed Talend (v 4.2.3) services to a windows 2003 server.
Everything was running well for about a week until our jobs stop running. Windows services status were seen "Started". The jobs resume after we restarted the services. However, such downtime is very disruptive for our company business.
Anyone could share what could have caused the problem?

administration log:
2012-05-11 01:40:18 WARN TalendJobStoreTX - Failed to override connection auto commit/transaction isolation.
org.h2.jdbc.JdbcSQLException: Database is already closed (to disable automatic closing at VM shutdown, add ";DB_CLOSE_ON_EXIT=FALSE" to the db URL)
at org.h2.message.DbException.getJdbcSQLException(DbException.java:327)
at org.h2.message.DbException.get(DbException.java:167)
at org.h2.message.DbException.get(DbException.java:144)
at org.h2.message.DbException.get(DbException.java:133)
at org.h2.jdbc.JdbcConnection.checkClosed(JdbcConnection.java:1348)
at org.h2.jdbc.JdbcConnection.checkClosed(JdbcConnection.java:1323)
at org.h2.jdbc.JdbcConnection.getAutoCommit(JdbcConnection.java:386)
at com.mchange.v2.c3p0.impl.NewProxyConnection.getAutoCommit(NewProxyConnection.java:985)
at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.hibernate.jdbc.BorrowedConnectionProxy.invoke(BorrowedConnectionProxy.java:40)
at $Proxy0.getAutoCommit(Unknown Source)
at org.quartz.impl.jdbcjobstore.AttributeRestoringConnectionInvocationHandler.setAutoCommit(AttributeRestoringConnectionInvocationHandler.java:83)
at org.quartz.impl.jdbcjobstore.AttributeRestoringConnectionInvocationHandler.invoke(AttributeRestoringConnectionInvocationHandler.java:65)
at $Proxy2.setAutoCommit(Unknown Source)
at org.quartz.impl.jdbcjobstore.JobStoreSupport.getConnection(JobStoreSupport.java:708)
at org.quartz.impl.jdbcjobstore.JobStoreTX.getNonManagedTXConnection(JobStoreTX.java:69)
at org.quartz.impl.jdbcjobstore.JobStoreSupport.doRecoverMisfires(JobStoreSupport.java:3103)
at org.quartz.impl.jdbcjobstore.JobStoreSupport$MisfireHandler.manage(JobStoreSupport.java:3869)
at org.quartz.impl.jdbcjobstore.JobStoreSupport$MisfireHandler.run(JobStoreSupport.java:3889)
2012-05-11 01:40:18 WARN TalendJobStoreTX - Failed to override connection auto commit/transaction isolation.
org.h2.jdbc.JdbcSQLException: Database is already closed (to disable automatic closing at VM shutdown, add ";DB_CLOSE_ON_EXIT=FALSE" to the db URL)
at org.h2.message.DbException.getJdbcSQLException(DbException.java:327)
at org.h2.message.DbException.get(DbException.java:167)
at org.h2.message.DbException.get(DbException.java:144)
at org.h2.message.DbException.get(DbException.java:133)
at org.h2.jdbc.JdbcConnection.checkClosed(JdbcConnection.java:1348)
at org.h2.jdbc.JdbcConnection.checkClosed(JdbcConnection.java:1323)
at org.h2.jdbc.JdbcConnection.getAutoCommit(JdbcConnection.java:386)
at com.mchange.v2.c3p0.impl.NewProxyConnection.getAutoCommit(NewProxyConnection.java:985)
at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.hibernate.jdbc.BorrowedConnectionProxy.invoke(BorrowedConnectionProxy.java:40)
at $Proxy0.getAutoCommit(Unknown Source)
at org.quartz.impl.jdbcjobstore.AttributeRestoringConnectionInvocationHandler.setAutoCommit(AttributeRestoringConnectionInvocationHandler.java:83)
at org.quartz.impl.jdbcjobstore.AttributeRestoringConnectionInvocationHandler.invoke(AttributeRestoringConnectionInvocationHandler.java:65)
at $Proxy2.setAutoCommit(Unknown Source)
at org.quartz.impl.jdbcjobstore.JobStoreSupport.getConnection(JobStoreSupport.java:708)
at org.quartz.impl.jdbcjobstore.JobStoreTX.getNonManagedTXConnection(JobStoreTX.java:69)
at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3739)
at org.quartz.impl.jdbcjobstore.JobStoreTX.executeInLock(JobStoreTX.java:90)
at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeWithoutLock(JobStoreSupport.java:3647)
at org.quartz.impl.jdbcjobstore.JobStoreSupport.getJobNames(JobStoreSupport.java:1912)
at org.quartz.core.QuartzScheduler.getJobNames(QuartzScheduler.java:1264)
at org.quartz.impl.StdScheduler.getJobNames(StdScheduler.java:505)
at org.talend.administrator.scheduler.business.SchedulerInitializer.shutdown(SchedulerInitializer.java:223)
at org.talend.administrator.scheduler.servlet.SchedulerInitializerServlet.destroy(SchedulerInitializerServlet.java:71)
at org.apache.catalina.core.StandardWrapper.unload(StandardWrapper.java:1394)
at org.apache.catalina.core.StandardWrapper.stop(StandardWrapper.java:1739)
at org.apache.catalina.core.StandardContext.stop(StandardContext.java:4764)
at org.apache.catalina.core.ContainerBase.removeChild(ContainerBase.java:924)
at org.apache.catalina.startup.HostConfig.undeployApps(HostConfig.java:1319)
at org.apache.catalina.startup.HostConfig.stop(HostConfig.java:1290)
at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:323)
at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:119)
at org.apache.catalina.core.ContainerBase.stop(ContainerBase.java:1086)
at org.apache.catalina.core.ContainerBase.stop(ContainerBase.java:1098)
at org.apache.catalina.core.StandardEngine.stop(StandardEngine.java:450)
at org.apache.catalina.core.StandardService.stop(StandardService.java:587)
at org.apache.catalina.core.StandardServer.stop(StandardServer.java:744)
at org.apache.catalina.startup.Catalina.stop(Catalina.java:648)
at org.apache.catalina.startup.Catalina$CatalinaShutdownHook.run(Catalina.java:692)
2012-05-11 01:40:18 ERROR QuartzSchedulerThread - quartzSchedulerThreadLoop: RuntimeException null
java.lang.reflect.UndeclaredThrowableException
at $Proxy2.rollback(Unknown Source)
at org.quartz.impl.jdbcjobstore.JobStoreSupport.rollbackConnection(JobStoreSupport.java:3583)
at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3746)
at org.quartz.impl.jdbcjobstore.JobStoreSupport.acquireNextTrigger(JobStoreSupport.java:2724)
at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:263)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.quartz.impl.jdbcjobstore.AttributeRestoringConnectionInvocationHandler.invoke(AttributeRestoringConnectionInvocationHandler.java:71)
... 5 more
Caused by: org.h2.jdbc.JdbcSQLException: Database is already closed (to disable automatic closing at VM shutdown, add ";DB_CLOSE_ON_EXIT=FALSE" to the db URL)
at org.h2.message.DbException.getJdbcSQLException(DbException.java:327)
at org.h2.message.DbException.get(DbException.java:167)
at org.h2.message.DbException.get(DbException.java:144)
at org.h2.message.DbException.get(DbException.java:133)
at org.h2.jdbc.JdbcConnection.checkClosed(JdbcConnection.java:1348)
at org.h2.jdbc.JdbcConnection.checkClosedForWrite(JdbcConnection.java:1333)
at org.h2.jdbc.JdbcConnection.rollback(JdbcConnection.java:435)
at com.mchange.v2.c3p0.impl.NewProxyConnection.rollback(NewProxyConnection.java:855)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.hibernate.jdbc.BorrowedConnectionProxy.invoke(BorrowedConnectionProxy.java:40)
at $Proxy0.rollback(Unknown Source)
... 10 more
2012-05-11 01:40:18 ERROR TalendJobStoreTX - MisfireHandler: Error handling misfires: null
java.lang.reflect.UndeclaredThrowableException
at $Proxy2.rollback(Unknown Source)
at org.quartz.impl.jdbcjobstore.JobStoreSupport.rollbackConnection(JobStoreSupport.java:3583)
at org.quartz.impl.jdbcjobstore.JobStoreSupport.doRecoverMisfires(JobStoreSupport.java:3133)
at org.quartz.impl.jdbcjobstore.JobStoreSupport$MisfireHandler.manage(JobStoreSupport.java:3869)
at org.quartz.impl.jdbcjobstore.JobStoreSupport$MisfireHandler.run(JobStoreSupport.java:3889)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.quartz.impl.jdbcjobstore.AttributeRestoringConnectionInvocationHandler.invoke(AttributeRestoringConnectionInvocationHandler.java:71)
... 5 more
Caused by: org.h2.jdbc.JdbcSQLException: Database is already closed (to disable automatic closing at VM shutdown, add ";DB_CLOSE_ON_EXIT=FALSE" to the db URL)
at org.h2.message.DbException.getJdbcSQLException(DbException.java:327)
at org.h2.message.DbException.get(DbException.java:167)
at org.h2.message.DbException.get(DbException.java:144)
at org.h2.message.DbException.get(DbException.java:133)
at org.h2.jdbc.JdbcConnection.checkClosed(JdbcConnection.java:1348)
at org.h2.jdbc.JdbcConnection.checkClosedForWrite(JdbcConnection.java:1333)
at org.h2.jdbc.JdbcConnection.rollback(JdbcConnection.java:435)
at com.mchange.v2.c3p0.impl.NewProxyConnection.rollback(NewProxyConnection.java:855)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.hibernate.jdbc.BorrowedConnectionProxy.invoke(BorrowedConnectionProxy.java:40)
at $Proxy0.rollback(Unknown Source)
... 10 more
2012-05-11 01:40:18 ERROR SchedulerInitializer - java.lang.reflect.UndeclaredThrowableException
2012-05-11 01:40:18 ERROR JDBCExceptionReporter - Database is already closed (to disable automatic closing at VM shutdown, add ";DB_CLOSE_ON_EXIT=FALSE" to the db URL)
2012-05-11 01:40:18 ERROR ExecutionTaskRefresher - ???Cannot create session or transaction.???
org.talend.administrator.common.exception.DBException: ???Cannot create session or transaction.???
at org.talend.administrator.common.persistence.hibernate.HibernatePersistenceAdapter.retrieve(HibernatePersistenceAdapter.java:573)
at org.talend.administrator.scheduler.persistence.helper.ExecutionTaskHelper.listExecutionTasks(ExecutionTaskHelper.java:245)
at org.talend.administrator.scheduler.business.ExecutionTaskRefresher$Refresher.runProcess(ExecutionTaskRefresher.java:231)
at org.talend.administrator.scheduler.business.ExecutionTaskRefresher$Refresher.run(ExecutionTaskRefresher.java:221)
Caused by: org.talend.administrator.common.exception.DBException: ???Cannot create session or transaction.???
at org.talend.administrator.common.persistence.hibernate.HibernatePersistenceAdapter.createTransaction(HibernatePersistenceAdapter.java:122)
at org.talend.administrator.common.persistence.hibernate.HibernatePersistenceAdapter.retrieve(HibernatePersistenceAdapter.java:561)
... 3 more
Caused by: org.talend.administrator.common.exception.DBException: ???Cannot create session or transaction.???
at org.talend.administrator.remoterepositorymgt.persistence.helper.UserHelper.countActiveUsers(UserHelper.java:200)
at org.talend.administrator.common.config.DbChecker.isDbOk(DbChecker.java:97)
at org.talend.administrator.common.config.DbChecker.checkDbConnectionWithTestRetrieve(DbChecker.java:127)
at org.talend.administrator.common.persistence.hibernate.HibernateTransactionHolder.checkDatabaseUsable(HibernateTransactionHolder.java:133)
at org.talend.administrator.common.persistence.hibernate.HibernateTransactionHolder.beginTransaction(HibernateTransactionHolder.java:171)
at org.talend.administrator.common.persistence.hibernate.HibernateTransactionHolder.<init>(HibernateTransactionHolder.java:47)
at org.talend.administrator.common.persistence.hibernate.HibernatePersistenceAdapter.createTransaction(HibernatePersistenceAdapter.java:115)
... 4 more
Caused by: org.talend.administrator.common.exception.DBException: ???Cannot create session or transaction.???
at org.talend.administrator.common.persistence.hibernate.HibernatePersistenceAdapter.createTransaction(HibernatePersistenceAdapter.java:122)
at org.talend.administrator.remoterepositorymgt.persistence.helper.UserHelper.countActiveUsers(UserHelper.java:183)
... 10 more
Caused by: org.talend.administrator.common.exception.DBException: ???Cannot commit transaction.???
at org.talend.administrator.common.persistence.hibernate.HibernateTransactionHolder.beginTransaction(HibernateTransactionHolder.java:172)
at org.talend.administrator.common.persistence.hibernate.HibernateTransactionHolder.<init>(HibernateTransactionHolder.java:47)
at org.talend.administrator.common.persistence.hibernate.HibernatePersistenceAdapter.createTransaction(HibernatePersistenceAdapter.java:115)
... 11 more
Caused by: org.hibernate.exception.GenericJDBCException: Cannot open connection
at org.hibernate.exception.SQLStateConverter.handledNonSpecificException(SQLStateConverter.java:103)
at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:91)
at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:43)
at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:29)
at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:420)
at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:144)
at org.hibernate.jdbc.JDBCContext.connection(JDBCContext.java:119)
at org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:57)
at org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1326)
at org.talend.administrator.common.persistence.hibernate.HibernateTransactionHolder.beginTransaction(HibernateTransactionHolder.java:166)
... 13 more
Caused by: org.h2.jdbc.JdbcSQLException: Database is already closed (to disable automatic closing at VM shutdown, add ";DB_CLOSE_ON_EXIT=FALSE" to the db URL)
at org.h2.message.DbException.getJdbcSQLException(DbException.java:327)
at org.h2.message.DbException.get(DbException.java:167)
at org.h2.message.DbException.get(DbException.java:144)
at org.h2.message.DbException.get(DbException.java:133)
at org.h2.jdbc.JdbcConnection.checkClosed(JdbcConnection.java:1348)
at org.h2.jdbc.JdbcConnection.checkClosed(JdbcConnection.java:1323)
at org.h2.jdbc.JdbcConnection.getAutoCommit(JdbcConnection.java:386)
at com.mchange.v2.c3p0.impl.NewProxyConnection.getAutoCommit(NewProxyConnection.java:985)
at org.hibernate.connection.C3P0ConnectionProvider.getConnection(C3P0ConnectionProvider.java:37)
at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:417)
... 18 more
5 REPLIES
One Star

Re: Talend Jobs stop running for unknown reason

Hi
Welcome to Talend Community!
Which version do you use, Commercial version or Community version?
It seems that you are using TAC and Job Server to deploy jobs.
Here are two issue which have been fixed on JIRA.
http://jira.talendforge.org/browse/TDI-9211
http://jira.talendforge.org/browse/TDI-9546
Regards.
Pedro
One Star

Re: Talend Jobs stop running for unknown reason

Thanks for your quick response.
We are using commercial version TIS 4.2.3. Both the links do not really relate to the message logs above. Moreover these fix are in place in v4.2.2.
One Star

Re: Talend Jobs stop running for unknown reason

Did the server reboot after applying windows updates? Did the database serviced restart if it rebooted?
One Star

Re: Talend Jobs stop running for unknown reason

There is no server reboot though there is one prompting for one.
One Star

Re: Talend Jobs stop running for unknown reason

I think we have found the issue. There were some backup scripts that is causing some I/O problem.