[resolved] Major Scheduler issue: "Awaiting Execution"
We are having an issue while scheduling processes in our production environment: (TIS 2.4.1 16077)
There are 2 jobs scheduled at 12:05 am and 1:35 am everyday. Almost every day I see a "Unexpected error, contact your administrator" on the job conductor. Trying to run the job manually brings up an icon with a '1 awaiting exec' message. Looking through the logs for when the job tries to execute:
2009-02-18 00:05:00 INFO RemoteJobExecution - ###### Task STARTED - ExecutionTask, contextParams=null]
2009-02-18 00:05:00 ERROR JDBCExceptionReporter - Communications link failure
Last packet sent to the server was 0 ms ago.
2009-02-18 00:05:00 ERROR JDBCExceptionReporter - Communications link failure
Last packet sent to the server was 1 ms ago.
2009-02-18 00:05:00 ERROR JDBCTransaction - JDBC commit failed
com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Connection.close() has already been called. Invalid operation in this state.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:353)
at com.mysql.jdbc.Util.getInstance(Util.java:336)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:985)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:957)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:927)
at com.mysql.jdbc.Connection.getMutex(Connection.java:2906)
at com.mysql.jdbc.Connection.commit(Connection.java:1466)
at org.hibernate.transaction.JDBCTransaction.commitAndResetAutoCommit(JDBCTransaction.java:139)
at org.hibernate.transaction.JDBCTransaction.commit(JDBCTransaction.java:115)
at org.talend.administrator.common.persistence.hibernate.HibernateTransactionHolder.commitWithFlush(HibernateTransactionHolder.java:78)
at org.talend.administrator.common.persistence.hibernate.HibernateTransactionHolder.commitAndClose(HibernateTransactionHolder.java:47)
at org.talend.administrator.common.persistence.hibernate.HibernatePersistenceAdapter.retrieve(HibernatePersistenceAdapter.java:414)
at org.talend.administrator.common.persistence.hibernate.HibernatePersistenceAdapter.retrieve(HibernatePersistenceAdapter.java:367)
at org.talend.administrator.common.persistence.hibernate.HibernatePersistenceAdapter.retrieve(HibernatePersistenceAdapter.java:333)
at org.talend.administrator.scheduler.persistence.helper.ExecutionTaskHelper.listExecutionTasks(ExecutionTaskHelper.java:192)
at org.talend.administrator.scheduler.business.ExecutionTaskRefresher$Refresher.run(ExecutionTaskRefresher.java:173)
Then:
2009-02-18 00:05:00 ERROR ExecutionTaskRefresher - org.talend.administrator.common.exception.DBException: ???Cannot commit transaction.???
After a few more cannot commit's/could not execute query:
2009-02-18 00:05:00 ERROR ExecutionTaskRefresher - No CurrentSessionContext configured!
org.hibernate.HibernateException: No CurrentSessionContext configured!
at org.hibernate.impl.SessionFactoryImpl.getCurrentSession(SessionFactoryImpl.java:540)
at org.talend.administrator.scheduler.business.ExecutionTaskRefresher.waitForDatabaseConnectionValid(ExecutionTaskRefresher.java:281)
at org.talend.administrator.scheduler.business.ExecutionTaskRefresher$Refresher.run(ExecutionTaskRefresher.java:204)
2009-02-18 00:05:00 ERROR RemoteJobExecution - ###### Exception on: ExecutionTask, contextParams=null]
org.talend.administrator.common.exception.SystemException: org.talend.administrator.common.exception.SystemException: org.hibernate.exception.JDBCConnectionException: could not execute query
Then this exception every 5 seconds:
2009-02-18 00:05:00 ERROR ErrorLogger - An error occured while marking executed job complete. job= 'DEFAULT.6'
org.quartz.JobPersistenceException: Failed to obtain DB connection from data source 'QRTZ_DS': org.hibernate.SessionException: Session is closed!
at org.quartz.impl.jdbcjobstore.JobStoreSupport.getConnection(JobStoreSupport.java:640)
at org.quartz.impl.jdbcjobstore.JobStoreTX.getNonManagedTXConnection(JobStoreTX.java:72)
at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3659)
at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3624)
at org.quartz.impl.jdbcjobstore.JobStoreSupport.triggeredJobComplete(JobStoreSupport.java:2868)
at org.quartz.core.QuartzScheduler.notifyJobStoreJobComplete(QuartzScheduler.java:1698)
at org.quartz.core.JobRunShell.run(JobRunShell.java:273)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:529)
Caused by: org.hibernate.SessionException: Session is closed!
at org.hibernate.impl.AbstractSessionImpl.errorIfClosed(AbstractSessionImpl.java:49)
at org.hibernate.impl.SessionImpl.connection(SessionImpl.java:358)
at org.talend.administrator.scheduler.QuartzConnectionProvider.getConnection(QuartzConnectionProvider.java:60)
at org.quartz.utils.DBConnectionManager.getConnection(DBConnectionManager.java:112)
at org.quartz.impl.jdbcjobstore.JobStoreSupport.getConnection(JobStoreSupport.java:633)
... 7 more
This has become a critical issue with our deployment. The only way I can find to fix it is to restart the Web server. I found an old bug report that related to this, it was suggested to add database.c3p0maxenabled=true to database.properties but i have done this and it did not work. Any help here would be appreciated
We are on 4.0.1 still. I found the problem in the bug tracker though:
http://www.talendforge.org/bugs/view.php?id=15630 I had installed the DQ .war file on tomcat, without removing the old TIS one.
After looking through my Scheduler logs, I have some of the same errors occurring. It seems to occur when I have two or more jobs executing at the same time. It happens sporadically.
Hi there,
Has there been a solution to this problem? I've started getting it in our environment as well. If I restart the whole lot, tomcat, jobserver, commandline etc then it runs, but the little icon stays on "waiting for task to finish" even though it has - the transaction has been committed etc. Which means, subsequent jobs "misfire" because it still thinks it's busy.
Thanks,
Hi rannandale
Which version are you using? For some crash sometimes, the previous execution does not finish as expected, so the next execution still wait for the last one to finish, need to restart tomcat.
Best regards
Shong
We are on 4.0.1 still. I found the problem in the bug tracker though:
http://www.talendforge.org/bugs/view.php?id=15630 I had installed the DQ .war file on tomcat, without removing the old TIS one.
