Qlik Community

Qlik Architecture Deep Dive Blog

Deep dives into specific back-end technologies which allow for the extension of Qlik to fit the needs of the enterprise.

Employee
Employee

External program tasks in Qlik Sense

External program tasks in Qlik Sense are simply another task type that exists, but not fully exposed by the QMC. They are tasks that are triggered from the QMC but allow for commands to be run in a command shell process behind the scenes; which in turns means you can run more or less anything you can run on the command prompt.

External program tasks in Qlik Sense are simply another task type that exists, but not fully exposed by the QMC. They are tasks that are triggered from the QMC but allow for commands to be run in a command shell process behind the scenes; which in turns means you can run more or less anything you can run on the command prompt. Once you define your command and create a new task, you can both trigger and chain these types of tasks in the QMC.

I will be using JavaScript for all the examples which means that node.js needs to be in your PATH variable (the location of node.exe). Again, you can run any command that you can run in the windows command prompt, so these examples are only for demo purposes.

Before we get started, we need to understand that IF you are calling scripts in your command, the script needs to be accessible by the user running the Qlik services. In this example, there’s a folder on the root C: drive called ‘externalTasksExample’, in this folder there is a createFile.js file which simply creates a new file in the same folder.

The file looks like:

var fs = require('fs');
var outputText = "This text will be in my output file.";
const data = new Uint8Array(Buffer.from(outputText));
fs.writeFile('C:\\externalTasksExample\\testCreateFile.txt', data, function (err) {
  if (err) throw err;
  console.log('The file has been saved!');
});

You can run the file from the command prompt by doing the following:

C:\externalTasksExample>“C:\Program Files\Qlik\Sense\ServiceDispatcher\Node\node.exe” createFile.js

When the command runs successfully, you should have a new file created in the externalTasksExample folder called “testCreateFile.txt” with the contents: “This text will be in my output file.”

Now we will delete this testCreateFile.txt and try to run that same JavaScript file using an external program task in Qlik Sense. The first thing we need to do is actually create the task.

The endpoint is:

POST /qrs/externalprogramtask

With the body being:

{
  "path": "C:\\Program Files\\Qlik\\Sense\\ServiceDispatcher\\Node\\node.exe",
  "parameters": "C:\\externalTasksExample\\index.js",
  "name": "ExampleExternalTask",
  "taskType": 1,
  "enabled": true,
  "taskSessionTimeout": 1440,
  "maxRetries": 0,
  "privileges": null,
  "schemaPath": "ExternalProgramTask"
}

Once you post this into Qlik, you should be able to go to the QMC and see the following task in the task list.

You should be able to select the task, click 'Start', and the same file should appear in 'C:\externalTasksExample'. You are now running an external task. Remember you don't have to use node.js, you can trigger anything you want as long as it can be run from the command line.

Scheduling

When it comes to scheduling an external program task, the easiest way is to create a "scheduled trigger" for a dummy reload task in the QMC, then move the trigger from the reload task to the external program task. You will need the ID of the external program task for this, which you would have gotten back in a response from the POST in the previous section, or you can use the GET call below to get a list of all external program tasks and their IDs.

GET /qrs/externalprogramtask

To start, go back to the Tasks list in the QMC and create a new task. Give it a memorable name (I will use 'tempTask'), and choose any application. Then under 'Triggers -> Actions', 'Create new scheduled trigger'. Fill out the scheduled trigger as if it was for the external program task.

Hit ok, click apply on the reload task, and you will then be taken to the 'Tasks' screen in the QMC. Next, we will use the API to find the new scheduled trigger we created.

GET /qrs/schemaevent?filter=name eq 'testTrigger'

Note the ID returned for this object, then get:

GET /qrs/schemaevent/{ID}

You will want to copy the entire response, which should look like something below.

{
    "id": "23268794-35f3-42f6-a9c0-57d91e2040cd",
    "createdDate": "2019-02-16T01:42:38.583Z",
    "modifiedDate": "2019-02-16T01:42:38.583Z",
    "modifiedByUserName": "DESKTOP-GS8LAA5\\jesse paris",
    "timeZone": "America/Chicago",
    "daylightSavingTime": 0,
    "startDate": "2019-02-15T19:47:30.000",
    "expirationDate": "9999-01-01T00:00:00.000",
    "schemaFilterDescription": [
        "* * - * * * * *"
    ],
    "incrementDescription": "0 0 0 0",
    "incrementOption": 0,
    "operational": {
        "id": "f113579d-3c68-4313-b448-1d934f09c998",
        "nextExecution": "2019-02-16T01:47:30.000Z",
        "timesTriggered": 0,
        "privileges": null
    },
    "name": "testTrigger",
    "enabled": true,
    "eventType": 0,
    "reloadTask": {
        "id": "4eb7f372-1333-4f1c-842d-678e6e3b3acb",
        "operational": {
            "id": "fe6cc117-5fcb-44bb-a18d-1c35dade5189",
            "lastExecutionResult": {
                "id": "9f4b69d7-c97e-4262-ab52-07401b93df7e",
                "executingNodeName": "desktop-gs8laa5",
                "status": 7,
                "startTime": "2019-02-16T00:53:17.140Z",
                "stopTime": "2019-02-16T00:53:19.171Z",
                "duration": 2031,
                "fileReferenceID": "792555fa-e96a-41ea-afbe-59b0028d8439",
                "scriptLogAvailable": false,
                "details": [
                    {
                        "id": "dc3a8eb0-9d44-4b32-aac7-bafb9c7d4e32",
                        "detailsType": 2,
                        "message": "Changing task state to Triggered",
                        "detailCreatedDate": "2019-02-16T00:53:17.140Z",
                        "privileges": null
                    },
                    {
                        "id": "8551a721-7d48-45fe-8505-c2fe6d1a1ee1",
                        "detailsType": 2,
                        "message": "Trying to start task. Sending task to slave scheduler desktop-gs8laa5",
                        "detailCreatedDate": "2019-02-16T00:53:17.468Z",
                        "privileges": null
                    },
                    {
                        "id": "55e9458f-af57-4f4b-92c5-463c9ad5ae67",
                        "detailsType": 2,
                        "message": "Changing task state from Triggered to Started",
                        "detailCreatedDate": "2019-02-16T00:53:17.718Z",
                        "privileges": null
                    },
                    {
                        "id": "5836b6c3-2287-4c8f-8c9b-af39dbea4176",
                        "detailsType": 2,
                        "message": "Reference to scriptlog added",
                        "detailCreatedDate": "2019-02-16T00:53:19.093Z",
                        "privileges": null
                    },
                    {
                        "id": "d0c66f1d-3238-43f5-a7c1-358a8d40a746",
                        "detailsType": 2,
                        "message": "Changing task state from Started to FinishedSuccess",
                        "detailCreatedDate": "2019-02-16T00:53:19.233Z",
                        "privileges": null
                    }
                ],
                "privileges": null
            },
            "nextExecution": "2019-02-16T01:47:30.000Z",
            "privileges": null
        },
        "name": "TestTaskDeleteME",
        "taskType": 0,
        "enabled": true,
        "taskSessionTimeout": 1440,
        "maxRetries": 0,
        "privileges": null
    },
    "userSyncTask": null,
    "externalProgramTask": null,
    "privileges": null,
    "schemaPath": "SchemaEvent"
}

We will take this response, remove the 'reloadTask' object, and add a reference to our external program task. Here we will need the ID that was returned when we created a new external program task. A final body should look like the following:

{
    "id": "23268794-35f3-42f6-a9c0-57d91e2040cd",
    "createdDate": "2019-02-16T01:42:38.583Z",
    "modifiedDate": "2019-02-16T01:42:38.583Z",
    "modifiedByUserName": "DESKTOP-GS8LAA5\\jesse paris",
    "timeZone": "America/Chicago",
    "daylightSavingTime": 0,
    "startDate": "2019-02-15T19:47:30.000",
    "expirationDate": "9999-01-01T00:00:00.000",
    "schemaFilterDescription": [
        "* * - * * * * *"
    ],
    "incrementDescription": "0 0 0 0",
    "incrementOption": 0,
    "operational": {
        "id": "f113579d-3c68-4313-b448-1d934f09c998",
        "nextExecution": "2019-02-16T01:47:30.000Z",
        "timesTriggered": 0,
        "privileges": null
    },
    "name": "testTrigger",
    "enabled": true,
    "eventType": 0,
    "reloadTask": null,
    "userSyncTask": null,
    "externalProgramTask": {
        "id": "e7ba25a8-a56c-4529-bc9a-af41755d3feb"
    },
    "privileges": null,
    "schemaPath": "SchemaEvent"
}

Using this body, you will update the schema event in the Qlik Sense Repository service with

PUT /qrs/schemaevent/{ID}

Once this is done, the task you created will now be assigned to the external program task. If you go back to the temporary reload task you created, you will notice that it no longer has a scheduled trigger associated to it, and you can delete that task. Any changed to the trigger need to be made to the schema event through the API, and any changes to the task itself needs to be made to the external program task through the API.

Happy APIing!