The following error is showing for tasks that were hanging in queue:
No engines were available to process this task. You can try to run the task manually now. Run this task during a time when your processors are available.
Resolution
If you are running too many tasks in parallel, experiencing time out and tasks failed, please consider the following.
- Set max.deployed.flows to a higher value.
This value should be considered upon how much memory you can use for task executions.
This is the amount of executed tasks you have in parallel.
For more information on how to set this value, please check here: Running tasks in parallel on Remote Engine.
- If the time does not matter on how long it takes for all tasks to complete,
then consider having few job executions in parallel.
For example: If you have 80 plans running at the same time, consider creating 3 plans
running at the same time with all the jobs included.
- Consider using remote engines in a cluster, so you have more
remote engines that can be used to execute the tasks.
For more information on clusters, please check here: Creating Remote Engine clusters.
- Increase the amount of memory you have in the environment where the remote engine is placed,
so you can allocate more memory to the remote engine.
- Adjust the time when tasks execute, so timeout does not occur.
This may be the only solution if you have very limited amount of memory to be used.
Cause
Too many tasks are running at the same time, and since the tasks that were in queue timed out, it failed to execute.
Related Content
On a different note, please also review the steps to clean up the task logs on the Remote Engine
automatic-task-log-cleanup
Environment
Talend Cloud
