topic Re: Too Close Rebooting Confuses Scheduler? in Management & Governance

Too Close Rebooting Confuses Scheduler?

justindallasverizon — Thu, 06 Mar 2025 20:58:02 GMT

Hello Folk,

Today, we had an issue where no tasks were getting run or scheduled. It didn't matter if they were invoked from the QMC or a 3rd party REST service.

When I look into the scheduler logs, I see the following message in the logs "Scheduler is not an active manager "

When I look at the boot time for the two schedulers, I see that they both started up within 12 seconds of one another.

My question is: Can two close of a boot time between 1 Manger+Worker node and 1 Worker node cause issues in finding who the master server is?

Re: Too Close Rebooting Confuses Scheduler?

David_Friend — Thu, 06 Mar 2025 21:06:09 GMT

@justindallasverizon not sure on that, did they eventually start working on their own or did you have to do a server restart(s)?

Re: Too Close Rebooting Confuses Scheduler?

justindallasverizon — Thu, 06 Mar 2025 21:41:21 GMT

We restarted the schedulers on both servers, and it resolved..

Would it help to have both nodes as Manager + Worker or are we restricted to one Manager per setup?

Re: Too Close Rebooting Confuses Scheduler?

diegozecchini — Fri, 07 Mar 2025 07:08:47 GMT

Hi!
maybe if both the Manager+Worker node and the Worker node boot almost simultaneously (within 12 seconds in your case), they might enter a race condition where neither is properly elected as the active scheduler manager. Did you think to manually delay the startup of the Worker node or use a startup script to introduce a wait time?