Too Close Rebooting Confuses Scheduler?

[justindallasverizon]

Hello Folk,

Today, we had an issue where no tasks were getting run or scheduled.  It didn't matter if they were invoked from the QMC or a 3rd party REST service.

When I look into the scheduler logs, I see the following message in the logs "Scheduler is not an active manager "

When I look at the boot time for the two schedulers, I see that they both started up within 12 seconds of one another.

My question is: Can two close of a boot time between 1 Manger+Worker node and 1 Worker node cause issues in finding who the master server is?

[David_Friend]

@justindallasverizon not sure on that, did they eventually start working on their own or did you have to do a server restart(s)?

[justindallasverizon]

We restarted the schedulers on both servers, and it resolved..

Would it help to have both nodes as Manager + Worker or are we restricted to one Manager per setup?

[diegozecchini]

Hi!
maybe if both the Manager+Worker node and the Worker node boot almost simultaneously (within 12 seconds in your case), they might enter a race condition where neither is properly elected as the active scheduler manager. Did you think to manually delay the startup of the Worker node or use a startup script to introduce a wait time?