We have a Talend Remote Engine Cluster with one engine set to run up to 6 jobs concurrently. We have a plan that runs on the :30 every hour of every day. It consistently takes ~20 minutes, when it should take ~2 minutes most of the time. This because tasks get stuck with “Awaiting an available engine to be executed”, although the engine should be/is available for doing work.
As an example, at 15:30 the plan started.
The first step has 1 task, starts at 15:30:08, execution starts at 15:30:13, and finishes in 5 seconds at 15:30:18.
The second step has 3 tasks, starts at 15:30:18, the longest task takes 6 seconds and it is done at 15:30:24.
The third step also has 3 tasks, starts at 15:30:24, the longest task takes 8 seconds and it is done at 15:30:32.
The fourth step is where things go wrong. It has 16 tasks.
- The first 6 tasks queue at 15:30:32 and start by 15:30:35, the longest taking 8 seconds. All 6 have finished by 15:30:43. Awesome!
- I would expect the next 6 tasks to be queuing in as tasks release, so by 15:30:43 the next 6 are running.
- However, the next 6 tasks wait exactly 4 minutes from the start time of the previous 6 within this step. The next 6 start at 15:34:35. They all finish within 8 seconds - why did they wait for minutes to process in a flash?
- I would again expect the next 6 tasks to be queuing in as tasks release, so by 15:30:51 we are on to the next 6.
- However, the next 6 tasks wait exactly 4 minutes again from the previous 6 before they start running, so starting at 15:38:35 (instead of 15:30:51 if the next task within the step would start immediately once a resource is freed).
This occurs twice within the fourth step above, and again twice within the fifth step of our plan, costing us >16 minutes of just waiting when we want it to run and finish faster so we can run it more frequently and leverage our home grown queueing systems.
The fact that it is 4 minutes exactly indicates to me there’s a configuration setting somewhere – but I can’t find any documentation about this. Do you know where in the configuration this would be? Why would tasks within the same step of a plan not immediately run once one of the previous tasks completes? (but instead, put a 4 minute wait before turning up our next 6)
I should add that we are using a Windows Server 2016 running the Talend Remote Engine 2.5.0 Service. The service itself is running, it's just that the when there are more tasks within a step than max concurrent tasks, we end up waiting 4 minutes for the next batch of tasks within the same step to be picked up.
