Skip to main content
Announcements
Introducing Qlik Answers: A plug-and-play, Generative AI powered RAG solution. READ ALL ABOUT IT!
cancel
Showing results for 
Search instead for 
Did you mean: 
Ulonis
Contributor
Contributor

Random appearing "Unexpected Error" on automated Jobs trough TAC

Hi,

since a few months we get a randomly appearing "Unexpected Error" as job are getting their sheduled run triggered in TAC (Enviroment Version 7.3.1). The Error, at first once in a few weeks, is now appearing daylie.

Repeating that process either by another CRON Trigger or via manual "Run" does work after waiting some time. So the problem is disappearing by itself after a bit and reappearing randomly. We had this on different Jobs all in the same project. Other projects seem to work fine or just didn't got that wrong moment for the Error to appear.

The Error always is dropping lines like the following few in our technical.log from TAC. Jobserver, CMD line etc. don't seem to mention those errors.
I put them in the Spoiler to shorten the post. Each "new" Error got its first line highlighted. Things like paths and Job Names/ Trigger Names got cleaned before too. (%dummy%)

 

Spoiler
2024-02-13 12:17:00 WARN RepoCheckoutHelper - Timeout when waiting unlocked SVN path '%installpath%\tac\apache-tomcat\temp\Talend_Administrator_Checkout_org.talend.administrator\trunk':
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at org.talend.commons.utils.threading.Locker.waitForLock(Locker.java:730)
at org.talend.administrator.remoterepositorymgt.persistence.helper.RepoCheckoutHelper.getItems(RepoCheckoutHelper.java:719)
at org.talend.administrator.remoterepositorymgt.persistence.helper.RepoCheckoutHelper.addItems(RepoCheckoutHelper.java:586)
at org.talend.administrator.remoterepositorymgt.persistence.helper.RepoCheckoutHelper.addItems(RepoCheckoutHelper.java:576)
at org.talend.administrator.remoterepositorymgt.persistence.helper.RepoCheckoutHelper.getProcessItems(RepoCheckoutHelper.java:558)
at org.talend.administrator.remoterepositorymgt.persistence.helper.RepoCheckoutHelper.getProcessItems(RepoCheckoutHelper.java:548)
at org.talend.administrator.remoterepositorymgt.project.SvnProjectManager.getProcessItem(SvnProjectManager.java:182)
at org.talend.administrator.remoterepositorymgt.project.SvnProjectManager.getProcessItemWithNodes(SvnProjectManager.java:321)
at org.talend.administrator.scheduler.business.wrapper.task.ExecutionTaskWrapper.getProcessItemWithChildren(ExecutionTaskWrapper.java:1070)
at org.talend.administrator.taskresuming.business.ProcessItemLoader.createInstanceFromSvnRepository(ProcessItemLoader.java:77)
at org.talend.administrator.scheduler.business.ExecutionTaskHandler.checkJobNeedsToBeRegenerated(ExecutionTaskHandler.java:1767)
at org.talend.administrator.scheduler.business.ExecutionTaskHandler.generateJobScriptUnit(ExecutionTaskHandler.java:510)
at org.talend.administrator.scheduler.business.ExecutionTaskHandler.runTaskUnit(ExecutionTaskHandler.java:1187)
at org.talend.administrator.scheduler.jobs.RemoteTaskExecution.executeRemoteJob(RemoteTaskExecution.java:507)
at org.talend.administrator.scheduler.jobs.RemoteTaskExecution.execute(RemoteTaskExecution.java:245)
at org.talend.administrator.scheduler.jobs.RemoteExecutionSwitcher.execute(RemoteExecutionSwitcher.java:54)
at org.quartz.core.JobRunShell.run(JobRunShell.java:199)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:546)
2024-02-13 12:17:00 ERROR TaskListenerRegister -
java.lang.NullPointerException
2024-02-13 12:17:00 ERROR RemoteTaskExecution - ###### Exception 'class org.talend.administrator.scheduler.exception.ExecutionTaskException': TAC cannot retrieve process Items please check your repository integrity or contact admin:ExecutionTask[taskName='%Job with Error%', taskId=261, quartzJobName='63', quartzFullTriggerName=JOB_CONDUCTOR.30, ownerSchedulerInstanceId=NON_CLUSTERED, server=localhost:8000, CronTrigger[talendTriggerName='%Trigger with Error%', talendTriggerId=181', idQuartzTrigger=30], contextParams=null]
org.talend.administrator.scheduler.exception.ExecutionTaskException: TAC cannot retrieve process Items please check your repository integrity or contact admin
at org.talend.administrator.scheduler.business.ExecutionTaskHandler.checkJobNeedsToBeRegenerated(ExecutionTaskHandler.java:1805)
at org.talend.administrator.scheduler.business.ExecutionTaskHandler.generateJobScriptUnit(ExecutionTaskHandler.java:510)
at org.talend.administrator.scheduler.business.ExecutionTaskHandler.runTaskUnit(ExecutionTaskHandler.java:1187)
at org.talend.administrator.scheduler.jobs.RemoteTaskExecution.executeRemoteJob(RemoteTaskExecution.java:507)
at org.talend.administrator.scheduler.jobs.RemoteTaskExecution.execute(RemoteTaskExecution.java:245)
at org.talend.administrator.scheduler.jobs.RemoteExecutionSwitcher.execute(RemoteExecutionSwitcher.java:54)
at org.quartz.core.JobRunShell.run(JobRunShell.java:199)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:546)
2024-02-13 12:17:00 ERROR RemoteTaskExecution - ###### Unexpected exception on: ExecutionTask[taskName='%Job with Error%', taskId=261, quartzJobName='63', quartzFullTriggerName=JOB_CONDUCTOR.30, ownerSchedulerInstanceId=NON_CLUSTERED, server=localhost:8000, CronTrigger[talendTriggerName='%Trigger with Error%', talendTriggerId=181', idQuartzTrigger=30], contextParams=null]
org.quartz.JobExecutionException: org.talend.administrator.scheduler.exception.ExecutionTaskException: TAC cannot retrieve process Items please check your repository integrity or contact admin [See nested exception: org.talend.administrator.scheduler.exception.ExecutionTaskException: TAC cannot retrieve process Items please check your repository integrity or contact admin]
at org.talend.administrator.scheduler.jobs.RemoteTaskExecution.executeRemoteJob(RemoteTaskExecution.java:551)
at org.talend.administrator.scheduler.jobs.RemoteTaskExecution.execute(RemoteTaskExecution.java:245)
at org.talend.administrator.scheduler.jobs.RemoteExecutionSwitcher.execute(RemoteExecutionSwitcher.java:54)
at org.quartz.core.JobRunShell.run(JobRunShell.java:199)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:546)
Caused by: org.talend.administrator.scheduler.exception.ExecutionTaskException: TAC cannot retrieve process Items please check your repository integrity or contact admin
at org.talend.administrator.scheduler.business.ExecutionTaskHandler.checkJobNeedsToBeRegenerated(ExecutionTaskHandler.java:1805)
at org.talend.administrator.scheduler.business.ExecutionTaskHandler.generateJobScriptUnit(ExecutionTaskHandler.java:510)
at org.talend.administrator.scheduler.business.ExecutionTaskHandler.runTaskUnit(ExecutionTaskHandler.java:1187)
at org.talend.administrator.scheduler.jobs.RemoteTaskExecution.executeRemoteJob(RemoteTaskExecution.java:507)
... 4 more
2024-02-13 12:17:00 ERROR RemoteExecutionSwitcher - ###### Exception on: org.talend.administrator.scheduler.exception.ExecutionTaskException: TAC cannot retrieve process Items please check your repository integrity or contact admin
org.quartz.JobExecutionException: org.talend.administrator.scheduler.exception.ExecutionTaskException: TAC cannot retrieve process Items please check your repository integrity or contact admin [See nested exception: org.talend.administrator.scheduler.exception.ExecutionTaskException: TAC cannot retrieve process Items please check your repository integrity or contact admin]
at org.talend.administrator.scheduler.jobs.RemoteTaskExecution.executeRemoteJob(RemoteTaskExecution.java:551)
at org.talend.administrator.scheduler.jobs.RemoteTaskExecution.execute(RemoteTaskExecution.java:245)
at org.talend.administrator.scheduler.jobs.RemoteExecutionSwitcher.execute(RemoteExecutionSwitcher.java:54)
at org.quartz.core.JobRunShell.run(JobRunShell.java:199)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:546)
Caused by: org.talend.administrator.scheduler.exception.ExecutionTaskException: TAC cannot retrieve process Items please check your repository integrity or contact admin
at org.talend.administrator.scheduler.business.ExecutionTaskHandler.checkJobNeedsToBeRegenerated(ExecutionTaskHandler.java:1805)
at org.talend.administrator.scheduler.business.ExecutionTaskHandler.generateJobScriptUnit(ExecutionTaskHandler.java:510)
at org.talend.administrator.scheduler.business.ExecutionTaskHandler.runTaskUnit(ExecutionTaskHandler.java:1187)
at org.talend.administrator.scheduler.jobs.RemoteTaskExecution.executeRemoteJob(RemoteTaskExecution.java:507)
... 4 more
2024-02-13 13:40:00 ERROR SvnCheckoutHelper - Unable to delete old checkout for project in%install Path%\tac\apache-tomcat\temp\Talend_Administrator_Checkout_org.talend.administrator\trunk. Error: %install Path%\tac\apache-tomcat\temp\Talend_Administrator_Checkout_org.talend.administrator\trunk\.svn\pristine
org.talend.commons.exception.SystemException: Unable to delete old checkout for project in%install Path%\tac\apache-tomcat\temp\Talend_Administrator_Checkout_org.talend.administrator\trunk. Error: %install Path%\tac\apache-tomcat\temp\Talend_Administrator_Checkout_org.talend.administrator\trunk\.svn\pristine
at org.talend.administrator.remoterepositorymgt.persistence.helper.SvnCheckoutHelper.prepareWorkingDirectoryUnit(SvnCheckoutHelper.java:347)
at org.talend.administrator.remoterepositorymgt.persistence.helper.SvnCheckoutHelper.prepareWorkingDirectoryCmdSuite(SvnCheckoutHelper.java:297)
at org.talend.administrator.remoterepositorymgt.persistence.helper.SvnCheckoutHelper.prepareWorkingDirectory(SvnCheckoutHelper.java:255)
at org.talend.administrator.remoterepositorymgt.persistence.helper.RepoCheckoutHelper.getItems(RepoCheckoutHelper.java:728)
at org.talend.administrator.remoterepositorymgt.persistence.helper.RepoCheckoutHelper.addItems(RepoCheckoutHelper.java:586)
at org.talend.administrator.remoterepositorymgt.persistence.helper.RepoCheckoutHelper.addItems(RepoCheckoutHelper.java:574)
at org.talend.administrator.remoterepositorymgt.persistence.helper.RepoCheckoutHelper.getProcessItems(RepoCheckoutHelper.java:558)
at org.talend.administrator.remoterepositorymgt.persistence.helper.RepoCheckoutHelper.getProcessItems(RepoCheckoutHelper.java:548)
at org.talend.administrator.remoterepositorymgt.project.SvnProjectManager.getProcessItem(SvnProjectManager.java:182)
at org.talend.administrator.remoterepositorymgt.project.SvnProjectManager.getProcessItemWithNodes(SvnProjectManager.java:321)
at org.talend.administrator.scheduler.business.wrapper.task.ExecutionTaskWrapper.getProcessItemWithChildren(ExecutionTaskWrapper.java:1070)
at org.talend.administrator.taskresuming.business.ProcessItemLoader.createInstanceFromSvnRepository(ProcessItemLoader.java:77)
at org.talend.administrator.scheduler.business.ExecutionTaskHandler.checkJobNeedsToBeRegenerated(ExecutionTaskHandler.java:1767)
at org.talend.administrator.scheduler.business.ExecutionTaskHandler.generateJobScriptUnit(ExecutionTaskHandler.java:510)
at org.talend.administrator.scheduler.business.ExecutionTaskHandler.runTaskUnit(ExecutionTaskHandler.java:1187)
at org.talend.administrator.scheduler.jobs.RemoteTaskExecution.executeRemoteJob(RemoteTaskExecution.java:507)
at org.talend.administrator.scheduler.jobs.RemoteTaskExecution.execute(RemoteTaskExecution.java:245)
at org.talend.administrator.scheduler.jobs.RemoteExecutionSwitcher.execute(RemoteExecutionSwitcher.java:54)
at org.quartz.core.JobRunShell.run(JobRunShell.java:199)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:546)
2024-02-13 13:40:00 ERROR TaskListenerRegister -
java.lang.NullPointerException
2024-02-13 13:40:01 ERROR RemoteTaskExecution - ###### Exception 'class org.talend.administrator.scheduler.exception.ExecutionTaskException': TAC cannot retrieve process Items please check your repository integrity or contact admin:ExecutionTask[taskName='%Job with Error 2%', taskId=360, quartzJobName='113', quartzFullTriggerName=JOB_CONDUCTOR.49, ownerSchedulerInstanceId=NON_CLUSTERED, server=localhost:8000, CronTrigger[talendTriggerName='%Trigger with Error 2%', talendTriggerId=295', idQuartzTrigger=49], contextParams=null]
org.talend.administrator.scheduler.exception.ExecutionTaskException: TAC cannot retrieve process Items please check your repository integrity or contact admin
at org.talend.administrator.scheduler.business.ExecutionTaskHandler.checkJobNeedsToBeRegenerated(ExecutionTaskHandler.java:1805)
at org.talend.administrator.scheduler.business.ExecutionTaskHandler.generateJobScriptUnit(ExecutionTaskHandler.java:510)
at org.talend.administrator.scheduler.business.ExecutionTaskHandler.runTaskUnit(ExecutionTaskHandler.java:1187)
at org.talend.administrator.scheduler.jobs.RemoteTaskExecution.executeRemoteJob(RemoteTaskExecution.java:507)
at org.talend.administrator.scheduler.jobs.RemoteTaskExecution.execute(RemoteTaskExecution.java:245)
at org.talend.administrator.scheduler.jobs.RemoteExecutionSwitcher.execute(RemoteExecutionSwitcher.java:54)
at org.quartz.core.JobRunShell.run(JobRunShell.java:199)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:546)

 

Labels (2)
5 Replies
Xiaodi_Shi
Support
Support

Hello,

Are you getting this random issue after applying some patch?
Can you please try to clear the TAC cache,restart TAC and let us know if you still face issue?

If you still face issues please help us with latest TAC Technical logs and catalina .out logs after TAC cache clean-up and restart.

https://community.qlik.com/t5/Official-Support-Articles/How-to-clear-the-Talend-Administration-Cente...

Best regards

Sabrina

Ulonis
Contributor
Contributor
Author

Hello Sabrina,

thanks for the fast answer. I cleared the TAC Cache as suggested 2 days ago.

Sadly the error did appear again yesterday. I attached the latest TAC technical and Catalina Log as asked. (Tasks and Triggers are again cleaned into "%Task%" etc. )

Xiaodi_Shi
Support
Support

Hello,

I see you are getting SVN warnings in your technical log.

Please try to go to TAC database, and then in configuration table , have the column:
scheduler.conf.DataProjectRefreshFrequency
By default it is 300 seconds (which is 5 minutes). Since you don't have GIT projects, then we don't need so often to checkout. Please set this to '3600'.

Feel free to let me know if it helps for you.

Best regards

Sabrina

Ulonis
Contributor
Contributor
Author

Hello,

 

thank you for that suggestion. I edited the db Entry to fit your suggestion. This was around 5 days ago. Since then it seemed to work somewhat until yesterday. The unexpected error startet to appear again slowly but steady.

Today it impacted the whole system again and forced us to reboot the whole server as we were not able to stop the services.

Since then it seems to somewhat work but unexpected errors appear from time to time again on sheduled jobs.

I would have added the logs again but after a short lookup i found nothing new. The errors are identical to those i already attached. Only the timestamps differ.

Do you have another suggestion? As this is starting to interfere with our productive processes we need to solve sooner than later. Depending on the next attempt I probably need to open a support incident.

 

Xiaodi_Shi
Support
Support

Hello,

Could you please create a support case on talend support portal so that we could give you a remote assistance through support cycle with priority to see if it is a bug and a patch could help you.

https://login.talend.com/support-login.php

Best regards

Sabrina