No Record of Tasks Being Submitted-QS Nov 2023 Patch 15

[justindallasverizon]

Hello Folks,

We are on QS Nov 2023 Patch 15 with 2 environments.  Today and last week, I've noticed a problem.

Sometimes, tasks simply can't be submitted to the cluster.  In the QMC > Tasks, when I click "Start", the bottom pop-up saying "Task submitted" will show, but when I refresh, I don't see any tasks running.

We also have a 3rd party system that invokes tasks using the REST API.  I see no evidence of those tasks ever getting invoked in the cluster.

When this happened last week, it mysteriously self-resolved. So I have two questions.

1. Which part of the logs should I look at?  The Scheduler logs can be overwhelming

2. What could be going on with the server where it just drops tasks submissions.

Any help is greatly appreciated.

[justindallasverizon]

The system is NOT saturated by the way.

[henrikalmen]

Are tasks being executed but QMC will not update so that you can't see that they have started, or are tasks not started at all?

[justindallasverizon]

Tasks aren't being executed at all.

[henrikalmen]

I would start with setting "task execution log level" to debug, in qmc->schedulers->[each reload node]. Or perhaps not start with that. First check what the logging level is currently and if it's lower than Info I'd set it to Info and see what the logs says. After that I'd increase it to debug, if I hadn't found the issue yet.

Have you installed and configured the Reloads Monitor app?

[justindallasverizon]

Yes, we have the Reloads monitor app.  But I don't think it would help since there is no record of a reload even being requested.

[justindallasverizon]

I'm seeing the following "Scheduler is not an active manager" message in one of my scheduler's logs.  I'm going to post yet another question about that.

[henrikalmen]

Check your worker/scheduler settings in QMC -> Schedulers -> [nodename] -> Advanced -> Type.

Manager (can be one or more nodes, but at least one) handles the scheduling and sends tasks to worker nodes. Worker is a node (at least one) that performs the reloads.