Qlik Architecture Deep Dive Blog

[Levi_Turner]

A deep dive into the QRS Repository API with a call out to receiving notifications on 

[Levi_Turner]

This is a more in-depth post as a follow up to the overview post: Getting Notified from Qlik Sense

Inside of the Qlik Sense Repository, there is a Notification API endpoint which allows the Repository to do a push notification, in JSON. In this example, we will use it receive a notification on a reload failure, but a more sophisticated example of using this endpoint can be found in the Qlik Sense Event Driven Cross Site App Promoter Repository on the Enterprise Architecture GitHub.

Note: subscriptions created using the Notification API are transient. They will not exist after the Qlik Sense Repository Service restarts. For production integrations, a methodology for polling the Repository to determine whether it has restarted / has the handle will need to be created. In the Qlik Sense Event Driven Cross Site App Promoter project, a NotificationCreator service is used to ensure that the subscription is always present in the Repository Service.

Note: In this post we will use screenshots from Postman since it does a good job of visually displaying the elements of an API call. These calls can be made using any scripting / programming language which can issue RESTful API calls.

Overview of the Repository Notification API

When creating a notification handle, there are 3 key params:

1. Name
2. changeType
3. (optional) filter
4. (optional) PropertyName
5. (optional) Condition

The Name param refers to the Repository entity that a change subscription should be configured for. This can be ExecutionResult (when monitoring task progress), App (when monitoring changes to an App), or any path where there is a Repository API (/qrs/path/..).

For a full listing of endpoints which are exposed on your version of Qlik Sense, issue a GET /qrs/about/api/description API call.

The changeType param refers to the numeric value of the change. The current values as of Qlik Sense September 2018, which were discernible via a GET /qrs/about/openapi/main call, are:

{
   "name": "changetype",
   "in": "query",
   "required": false,
   "default": "Undefined",
   "type": "integer",
   "allowEmptyValue": true,
   "enum": [
       0,
       1,
       2,
       3
   ],
   "x-enumNames": [
       "Undefined",
       "Add",
       "Update",
       "Delete"
   ]
}

The filter param is optional, but generally encouraged. For the specific value, a close inspection of the spec for the Name which will be monitored is needed. Since this example is focusing on tasks, the ExecutionResult spec, which is discernible via a GET /qrs/about/openapi/main call, is as follows:

"ExecutionResult": {
    "type": "object",
    "properties": {
        "id": {
            "type": "string",
            "format": "uuid"
        },
        "createdDate": {
            "type": "string",
            "format": "date-time"
        },
        "modifiedDate": {
            "type": "string",
            "format": "date-time"
        },
        "modifiedByUserName": {
            "type": "string"
        },
        "schemaPath": {
            "type": "string"
        },
        "privileges": {
            "type": "array",
            "items": {
                "type": "string"
            }
        },
        "taskID": {
            "type": "string",
            "format": "uuid"
        },
        "executionID": {
            "type": "string",
            "format": "uuid"
        },
        "appID": {
            "type": "string",
            "format": "uuid"
        },
        "executingNodeID": {
            "type": "string",
            "format": "uuid"
        },
        "executingNodeName": {
            "type": "string"
        },
        "status": {
            "type": "integer",
            "enum": [
                0,
                1,
                2,
                3,
                4,
                5,
                6,
                7,
                8,
                9,
                10,
                11,
                12
            ],
            "x-enumNames": [
                "NeverStarted",
                "Triggered",
                "Started",
                "Queued",
                "AbortInitiated",
                "Aborting",
                "Aborted",
                "FinishedSuccess",
                "FinishedFail",
                "Skipped",
                "Retry",
                "Error",
                "Reset"
            ]
        },
        "startTime": {
            "type": "string",
            "format": "date-time"
        },
        "stopTime": {
            "type": "string",
            "format": "date-time"
        },
        "duration": {
            "type": "integer",
            "format": "milliseconds"
        },
        "fileReferenceID": {
            "type": "string",
            "format": "uuid"
        },
        "scriptLogAvailable": {
            "type": "boolean"
        },
        "details": {
            "type": "array",
            "items": {
                "$ref": "#/definitions/ExecutionResultDetailCondensed"
            }
        }
    },
    "x-qlik-stability": "Locked"
}

If we want to monitor on a task which fails due to a script related problem, then the status value that we are interested in is FinishedFail which has a numeric value of 8.

The PropertyName param is also optional. Much like the filter param, you are doing further drill downs. But instead of doing a filter on the entity specified in the Name, you are monitoring changes on the property of the entity without specifying a known value. For example, if you wanted monitor for when an app is published, you can create a subscription with this config:

This will provoke a response when:

• An app is published
• Any published App changes (e.g. it’s renamed)

For the use of monitoring the pure publish operation, there are other scenarios where you want covered:

• When an app is published via publish and replace
• When an app is moved to a new stream

To allow for coverage of those scenarios, we can create a subscription with the following config:

A clean distinction between PropertyName and Filter is that PropertyName is a reduction in the monitoring space but without unknown values. In the above scenario, we do not know what the publishTime will be, but merely that we want to monitor for changes in that element of the App’s entity. Filters, on the other hand, require known values.

The condition param likewise is optional. This param allows a script / API call to be issued and handle created only if the condition is fulfilled. When the conditional is not, fulfilled, a null GUID for the handle will be returned:

Reload failure specific work-flow

To continue with the specific work-flow on task failures, for the params section of the POST that we will construct to create the subscription is:

For the headers, we are using standard headers for a QRS API call:

 

For the body of the POST request, we will wrap a RESTful endpoint in double quotes:

 

In the 201 response from the Qlik Sense Repository Service we receive the Handle ID of the subscription:

{
    "value": "0a950014-61df-44d5-b97f-b911231aa746"
}

Note: Repeated POSTs using the same inputs will return the same handle value

Once this handle is created, our JSON listener receives this style of input on a task failure:

{
  "changeType": 2, 
  "objectType": "ExecutionResult", 
  "objectID": "783d83ba-436c-475c-993c-fdd0d6f47801", 
  "changedProperties": 
    [
    "modifiedDate", 
    "fileReferenceID"
    ], 
  "engineID": "", 
  "engineType": "", 
  "originatorNodeID": "f206bf30-1e5c-498a-8f73-f1f9e2120863", 
  "originatorHostName": "QLIKSENSESERVER", 
  "originatorContextID": None, 
  "createdDate": "2018-11-01T21:37:33.787Z", 
  "modifiedDate": "2018-11-01T21:37:34.811Z", 
  "schemaPath": "ExternalChangeInfo"
}

For a productionalized integration, we’d do a subsequent GET /qrs/executionresult/783d83ba-436c-475c-993c-fdd0d6f47801 call which would return the relevant details like the Task Name / App Name:

{
    "id": "783d83ba-436c-475c-993c-fdd0d6f47801",
    "createdDate": "2018-11-01T21:37:33.787Z",
    "modifiedDate": "2018-11-01T21:37:34.964Z",
    "modifiedByUserName": "INTERNAL\\sa_scheduler",
    "taskID": "9981ae16-a0fa-466e-a2c2-b8588eafe395",
    "executionID": "ef6ca442-ed5c-4df4-a9ad-bb52c339c496",
    "appID": "2015e38b-650c-42df-8539-099aa8fef445",
    "executingNodeID": "682ddb96-ad29-43b1-89c9-8c25d58ec064",
    "executingNodeName": "QlikSENSESERVER",
    "status": 8,
    "startTime": "2018-11-01T21:37:33.769Z",
    "stopTime": "2018-11-01T21:37:34.892Z",
    "duration": 1123,
    "fileReferenceID": "bf749515-ab2f-42ce-94f9-c698b9c57b42",
    "scriptLogAvailable": true,
    "details": [
        {
            "id": "b2987c70-2615-434f-904b-a72e7e8a44bb",
            "detailsType": 2,
            "message": "Message from ReloadProvider: Reload failed in Engine. Check engine or script logs.",
            "detailCreatedDate": "2018-11-01T21:37:34.826Z",
            "privileges": null
        },
        {
            "id": "432952f3-6f1f-4b1c-9efe-3e672dca0ed3",
            "detailsType": 2,
            "message": "Changing task state from Started to FinishedFail",
            "detailCreatedDate": "2018-11-01T21:37:34.827Z",
            "privileges": null
        },
        {
            "id": "ed2e16ca-781c-44a7-a438-5245e2f23a81",
            "detailsType": 2,
            "message": "Changing task state from Triggered to Started",
            "detailCreatedDate": "2018-11-01T21:37:34.087Z",
            "privileges": null
        },
        {
            "id": "990de3de-5d96-4bf9-89da-68c735c3489c",
            "detailsType": 2,
            "message": "Max retries reached (0)",
            "detailCreatedDate": "2018-11-01T21:37:34.892Z",
            "privileges": null
        },
        {
            "id": "009c6efc-ca33-44d3-b8f6-86396b441e6c",
            "detailsType": 2,
            "message": "Changing task state to Triggered",
            "detailCreatedDate": "2018-11-01T21:37:33.770Z",
            "privileges": null
        },
        {
            "id": "352f0b8e-e956-4b82-8b8b-826144cdad48",
            "detailsType": 2,
            "message": "Trying to start task. Sending task to slave scheduler QLIKSENSESERVER",
            "detailCreatedDate": "2018-11-01T21:37:33.998Z",
            "privileges": null
        },
        {
            "id": "16e3ec30-5ea1-4bd2-afe5-24b66ce9414f",
            "detailsType": 2,
            "message": "Reference to scriptlog added",
            "detailCreatedDate": "2018-11-01T21:37:34.779Z",
            "privileges": null
        }
    ],
    "privileges": null,
    "schemaPath": "ExecutionResult"
}

[saurau]

I am able to subscribe, however, the action is not getting triggered upon the App Publish event.

I have passed the API URL as a body in double-quotes which need to be triggered after app Publish.

I can't see the API trigger.

How can I troubleshoot and see notification API working and triggering action?

[Levi_Turner]

@saurau : What are your:

1. Name
2. changeType
3. (optional) filter
4. (optional) PropertyName
5. (optional) Condition

?

Here's an example which is monitoring on publish events: https://github.com/eapowertools/qs-event-driven-cross-site-app-promoter/blob/master/notification_cre...

[saurau]

Thanks @Levi_Turner , Here are the field details:

1. Name: App
2. changeType : 2
3. (optional) filter
4. (optional) PropertyName: publishTime
5. (optional) Condition

Quick question, can we get anywhere the list of PropertyName that we can incorporate here?

Thanks, I will look into the code and try to incorporate the same in my code to monitor why the listener URL is not working.

[Levi_Turner]

publishTime is an attribute of the entity which is being monitored (app in this instance). Example App structure:

  {
    "id": "674c8421-3ea4-4f78-a1cd-edf2c87e4945",
    "createdDate": "2019-03-01T04:14:50.979Z",
    "modifiedDate": "2020-05-07T12:20:00.715Z",
    "modifiedByUserName": "QTSEL\\ltu",
    "customProperties": [],
    "owner": {
      "id": "0d11d863-1c53-4a00-8699-3837b09109aa",
      "userId": "ltu",
      "userDirectory": "QTSEL",
      "name": "Levi Turner",
      "privileges": null
    },
    "name": "App Stream Example",
    "appId": "",
    "sourceAppId": "00000000-0000-0000-0000-000000000000",
    "targetAppId": "00000000-0000-0000-0000-000000000000",
    "publishTime": "2019-07-28T15:51:01.094Z",
    "published": true,
    "tags": [],
    "description": "Pass DocumentName() to /qrs/app/full to access the stream of the App",
    "stream": {
      "id": "c88f372b-7fa7-493e-92d8-d4caa0f2a4f1",
      "name": "Completed Code",
      "privileges": null
    },
    "fileSize": 153544,
    "lastReloadTime": "2019-03-01T04:36:22.691Z",
    "thumbnail": "",
    "savedInProductVersion": "12.421.3",
    "migrationHash": "21ecc792c56e18162f1785d3d41f28fdaced5c96",
    "dynamicColor": "",
    "availabilityStatus": 5,
    "privileges": null,
    "schemaPath": "App"
  }

[saurau]

I am still not able to figure it out about listener url trigger, My responses are:

https://<BASE_URL>/qrs/notification?name=app&changeType=update&propertyName=publishTime 
Response : {"value":"124fcb61-fa7b-4a09-840f-4b0bb8efcadc"} 
Status code : 201

My listener URL is different, however having in same domain, is it the issue? It could be certificate issue or any kind of other communication issue.

I have also got the response for notification handler as:

{"callbackUri":"https://<listener URL(removed as per privacy concern)>","typeName":"App","id":"00000000-0000-0000-0000-000000000000","filter":null,"changeType":2,"propertyName":"publishTime","startTime":"2021-01-11T12:03:41.732Z","handle":"8e5e446a-44f3-423b-9750-77a23004ffe9","clientId":"00000000-0000-0000-0000-000000000000","schemaPath":"RestChangeNotificationRegistration"}

That's why my question still relies it how to troubleshoot listener URL connectivity and see issues with it?