Qlik Engine Health-Check Analysis (app included)

[RadovanOresky]

Monitoring current performance and availability of Qlik Sense site is not out-of-the-box feature, but can be easily achieved with native REST connector and requesting engine\healthcheck JSON API.

There are other tools and approaches, but since I already have the best BI tool available 😉, I found it reasonable to use it. All that is necessary is just:

1. download and import the attached app
2. create REST Data Connections and adjust the settings in the script
3. schedule app reload

This Qlik Sense app:

• loads engine\healthcheck JSON data from multiple nodes
• parses and consolidates it to two tables
• stores them to QVDs with all previous snapshots
• visualizes key metrics of engine utilization and availability in time

It's recommended to deploy and schedule this app on DEV or TEST site, pointing to all production nodes. Otherwise, there will be no data loaded in case the production engine (or the whole server) crashes.
(just FYI - DEV and TEST sites are included in the professional license subscription)

It's also recommended to read the official help site info about the engine API before you deploy this app.

After importing the app to a suitable site, follow these steps:

1. Decide on target folder for result QVD files, where parsed snapshot of JSON data will be stored. Create a folder Data Connection (if it doesn’t already exist) and set its Lib name to vQVDpath variable in Settings script section.

2. Create a REST Data Connection to each node:

URL: https://<server domain>/engine/healthcheck OR https://<node IP address>:4747/engine/healthcheck

Authentication Schema: Windows NTLM
User name & password: type root admin or service user credentials
check "Skip server certificate validation" if there is no valid certificate on target node

Query headers:
User-Agent: Windows
Cache-Control: no-cache

3. Adjust the INLINE table in the Settings section of the script according to your site

node - name of the node that will be shown for app’s user
restConnection - name of the REST connection that points to node’s engine/healthcheck
RAM - amount of node's memory in GB
CPU - number of node's logical processors

4. Schedule a reload task. The recommended app reload frequency is 5 minutes or less (tested reload duration on the 4-node site is 12 seconds). The script will return "N/A" in [isSaturated] field when the engine is not responding.

The current roadmap includes - script for email notifications, possibility to load only N days back, mapping app name to IDs, node comparisons.

PLEASE! If you make adjustments or additions to the script or front-end visualizations, please share them back on the Qlik Community.

Happy Qliking 🙂
-RADO-

[RadovanOresky]

Hi @Ramajola,

I believe that the problem is with the INLINE table in the Settings section. Yours should look like this:

EngineNodes:
load * inline [
    node, restConnection, RAM, CPU
    Prod, REST_Healthcheck, 16, 8
    ];

 If you omit the first line you won't get any data.

It gives you an error that it cannot find the MainMeasures QVD file, because it wasn't created.

Also, you don't need to edit the variables at the end. The declaration serves only to clear them.

[Ramajola]

@RadovanOresky Thank you for all your assistance .

Have you a nice day! 

[RadovanOresky]

Hi @Mauritz_SA ,

I'm glad you find the app helpful 🙂

I would definitely set alerts on RAM saturation ( isSaturated = 'True' ) and then maybe the server uptime = how long from the last Qlik engine restart. From my experience, this should not be longer than 10 days, because the  Qlik cache is probably stale anyway and other issues can accumulate.

Also, I'm thinking of giving more attention to a difference between Cache Lookups and Cache Hits. A large difference can indicate that the cache is not very useful and lots of calculations have to be made, which can lead to performance problems.

I guess this can be also indicated with a ratio of cache Replaced to cache Added. But I'd need to research this a bit more.

What are your ideas?

[Mauritz_SA]

Hi @RadovanOresky 

Thanks for the feedback! My team and I will have a look at this during the next couple of weeks and I'll let you know what we find. We're more on the dev side than the server side, but I'll send our thoughts anyways. Really like the conceot of this app, well done!

Regards,

Mauritz

[userid128223]

I am getting following error:

Started loading data

Connected EngineNodes << a3b74156-dc93-4501-89ea-9dc3e49b61c4 Lines fetched: 1

Connected Rest_EngineHealthCheck << RestConnectorMasterTable mainMeasures << AUTOGENERATE(1) Lines fetched: 1 mainMeasures << EngineHealthCheck_MainMeasures (QVD (row-based) optimized) Lines fetched: 851 No Apps mainMeasures << EngineHealthCheck_MainMeasures Lines fetched: 851 apps << EngineHealthCheck_Apps Lines fetched: 0 _temp_linkTable << mainMeasures Lines fetched: 851 _temp_linkTable << apps Lines fetched: 10 linkTable << _temp_linkTable Lines fetched: 10 _join_max_timestamp_flag << linkTable Lines fetched: 1 Creating search index Search index creation completed successfully

The following error occurred:

QVX_UNEXPECTED_END_OF_DATA: Unexpected character encountered while parsing value: <. Path '', line 0, position 0.

[mjperreault]

Hello All,

When trying to create the REST connection to a rim node I am getting an Error "Connection with server closed prematurely" has anyone else seen this?

Thanks,
Mark

[RadovanOresky]

Hi @mjperreault,

The reason for this could be that the server which is trying to connect to engine\healthcheck endpoint on the RIM node cannot see the URL that you provided.

Try to change the server name in the URL based on what you have in the Virtual Proxy whitelist, or try to use IP address along with the 4747 port.

[iceman35]

Hi @RadovanOresky ,

I'm also getting the following error when trying to load Rest_EngineHealthCheck :

Unexpected character encountered while parsing value: <. Path '', line 0, position 0.

Thanks for your help.

Bastien

[RadovanOresky]

Hi @iceman35 , can you confirm that the URL that you set in the REST data connection gives you good response? The best way to test it is in Postman with other connections settings (query headers).

[santhoshkumar]

@RadovanOresky  thanks for this tool. just trying in my site which is enabled with header authentication not windows. By any chance we can make this work with header proxy? or windows is mandatory?