2 Replies Latest reply: Jan 30, 2018 8:15 AM by Rheiner Rhenus RSS

    Scheduler - Could not reserve an executor for task: No node found for task(name/id)

    Rheiner Rhenus

      Hello Everyone,

       

      we are running a multi-node Qlik Sense environment in our company and among other tasks there is one that is reloading

      a "real-time" app every 5 minutes. Almost every day, one execution of this task fails (at different times).

       

      The log files do not really contain any hint what is happening, but entries in the log_entries table of the Qlik database

      show the following errors

       

      {

          "Origin": "Not available",

          "ProxyPackageId": "0",

          "ObjectName": "RT Fehlmengenreport|RT Fehlmengenreport",

          "Description": "Command=Finished task;Result=65;ResultText=Error: Failed",

          "ObjectId": "b86d7663-525f-4249-bdbf-c4f46ab62765|d2047645-31af-4d47-8024-c3133d46bddb",

          "RequestSequenceId": "0",

          "UserDirectory": "INTERNAL",

          "UserId": "sa_scheduler",

          "Command": "Finished task",

          "Result": "65",

          "Context": "Not available",

          "Message": "Task finished with state Error",

          "Service": "Scheduler",

          "ProxySessionId": "0"

      }

       

      {

          "Status": "Triggered",

          "ProxySessionId": "0",

          "AppName": "RT Fehlmengenreport",

          "StartTime": "20180127T065958.552+0100",

          "ExecutionId": "",

          "StopTime": "17530101T010000.000+0100",

          "ExecutingNodeName": "",

          "User": "INTERNAL\\sa_scheduler",

          "TaskId": "b86d7663-525f-4249-bdbf-c4f46ab62765",

          "AppId": "d2047645-31af-4d47-8024-c3133d46bddb",

          "Duration": "0",

          "TaskName": "RT Fehlmengenreport",

          "ExecutingNodeId": "",

          "Message": "Could not reserve an executor for task: No node found for task(name/id) RT Fehlmengenreport/b86d7663-525f-4249-bdbf-c4f46ab62765",

          "FailureReason": ""

      }

       

      {

          "Status": "Error",

          "ProxySessionId": "0",

          "AppName": "RT Fehlmengenreport",

          "StartTime": "20180127T065958.552+0100",

          "ExecutionId": "",

          "StopTime": "17530101T010000.000+0100",

          "ExecutingNodeName": "",

          "User": "INTERNAL\\sa_scheduler",

          "TaskId": "b86d7663-525f-4249-bdbf-c4f46ab62765",

          "AppId": "d2047645-31af-4d47-8024-c3133d46bddb",

          "Duration": "0",

          "TaskName": "RT Fehlmengenreport",

          "ExecutingNodeId": "",

          "Message": "Failed to start session",

          "FailureReason": ""

      }

       

      {

          "Origin": "Not available",

          "ProxyPackageId": "0",

          "ObjectName": "RT Fehlmengenreport|RT Fehlmengenreport",

          "Description": "Command=Start task;Result=40;ResultText=Error: Failed",

          "ObjectId": "b86d7663-525f-4249-bdbf-c4f46ab62765|d2047645-31af-4d47-8024-c3133d46bddb",

          "RequestSequenceId": "0",

          "UserDirectory": "INTERNAL",

          "UserId": "sa_scheduler",

          "Command": "Start task",

          "Result": "40",

          "Context": "Not available",

          "Message": "No slaves found to execute task",

          "Service": "Scheduler",

          "ProxySessionId": "0"

      }

       

      I need to add that we have a central node (with a master scheduler) and two reload nodes (with slave schedulers). The "real-time" app is always reloaded on the same reload node because we created load balancing rules with custom properties to assure that.

       

      We would like to know more about the red marked error message from above ("Could not reserve an executor for task: No node found for task(name/id)"). Where is the executor reserved? How many executors are there? Is it configurable?

       

      If you happen to know something about it and can give us a little advice, it would be much appreciated.

       

      Thank you very much.

        • Re: Scheduler - Could not reserve an executor for task: No node found for task(name/id)
          Martin Sandin

          Hello Rheiner!

           

          Executor refers to the entity which can execute tasks (e.g. reloads), i.e. the slave schedulers. So this means that the one slave scheduler which is eligible according to your load balancing rules can't actually do it for some reason which could include engine problems, slave scheduler <-> engine connectivity problems, QRS connectivity problems, or master <-> slave connectivity problems. You could further examine the logs around the time when this happens on both the master and the slave in question.

            • Re: Scheduler - Could not reserve an executor for task: No node found for task(name/id)
              Rheiner Rhenus

              Hello Martin,

               

              thanks for your explanation. Examining the logs of the master scheduler helped us to identify the problem. To be precise looking for System.Scheduler.Scheduler.Master.SlaveManager Logger entries in the System_Scheduler log. It said that

              Slave-node xyz1.yy.zz cannot run task RT Fehlmengenreport. Message: Engine not alive. We are now investigating why the engine was not "ready" to do the reload on the slave node. But that is another topic - maybe has something to do with our ESX cluster that the Qlik nodes are running on ...

               

              By the way we also configured the task to do 3 retries - just in case.