Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.
We have a Talend Remote Engine Cluster with one engine set to run up to 6 jobs concurrently. We have a plan that runs on the :30 every hour of every day. It consistently takes ~20 minutes, when it should take ~2 minutes most of the time. This because tasks get stuck with “Awaiting an available engine to be executed”, although the engine should be/is available for doing work.
As an example, at 15:30 the plan started.
The first step has 1 task, starts at 15:30:08, execution starts at 15:30:13, and finishes in 5 seconds at 15:30:18.
The second step has 3 tasks, starts at 15:30:18, the longest task takes 6 seconds and it is done at 15:30:24.
The third step also has 3 tasks, starts at 15:30:24, the longest task takes 8 seconds and it is done at 15:30:32.
The fourth step is where things go wrong. It has 16 tasks.
This occurs twice within the fourth step above, and again twice within the fifth step of our plan, costing us >16 minutes of just waiting when we want it to run and finish faster so we can run it more frequently and leverage our home grown queueing systems.
The fact that it is 4 minutes exactly indicates to me there’s a configuration setting somewhere – but I can’t find any documentation about this. Do you know where in the configuration this would be? Why would tasks within the same step of a plan not immediately run once one of the previous tasks completes? (but instead, put a 4 minute wait before turning up our next 6)
I should add that we are using a Windows Server 2016 running the Talend Remote Engine 2.5.0 Service. The service itself is running, it's just that the when there are more tasks within a step than max concurrent tasks, we end up waiting 4 minutes for the next batch of tasks within the same step to be picked up.
Didn't mean to open separate reply - fwiw still occurring this morning, causing us to miss our SLAs.
We upgraded to the TalendRemoteEngine 2.8.4 and the issue is still occurring.
We upgraded to the TalendRemoteEngine 2.8.4 and the issue is still occurring.
Hi,
This isn't a helpful comment, but just insight, I've noticed that when you manually trigger a job that is scheduled, and the manual invocation is still running when the scheduled one starts, it will wait for 4 minutes before attempting to try again.
So, say I manually trigger a job that runs for 2 minutes at 15:59 (so runs 15:59-16:01) and the scheduled instance starts at 16:00....it will sit in the same waiting state until 16:04 before trying to start the job again. Which essentially sounds like the behaviour you are seeing. when one job in the plan finishes and goes to start the next, its as if the Remote Engine thinks its busy so makes it wait 4 mins. I'd imagine this is a setting up in the TMC that you dont have any control over.
Hi,
This isn't a helpful comment, but just insight, I've noticed that when you manually trigger a job that is scheduled, and the manual invocation is still running when the scheduled one starts, it will wait for 4 minutes before attempting to try again.
So, say I manually trigger a job that runs for 2 minutes at 15:59 (so runs 15:59-16:01) and the scheduled instance starts at 16:00....it will sit in the same waiting state until 16:04 before trying to start the job again. Which essentially sounds like the behaviour you are seeing. when one job in the plan finishes and goes to start the next, its as if the Remote Engine thinks its busy so makes it wait 4 mins. I'd imagine this is a setting up in the TMC that you dont have any control over.
Hi All ,
Did anyone come across any solution for this problem?
Thanks,
Naga