Qlik Community

Qlik Support Updates Blog

Important and useful support information about end-of-product support, new service releases, and general support topics.

Announcements
Announcement: Certain actions are currently causing a Page Not Found error. This is a known issue and we are working with the platform vendor to investigate and resolve it.

All you need to know about deploying the Qlik Sense Monitoring Applications

Image Not found

After installing Qlik Sense Enterprise, you probably have noticed that you have been provided with two applications respectively called Operation Monitor and License Monitor. But wait there are more…

Generally speaking, we call those applications Monitoring Applications

Today I would like to introduce you to those applications and how we setup and configure them.

Today I will try to answer and clarify the following question you may have

What and how many are they?

How does the Reload work?

Setup and configuration

 

What and how many are they?

In total, and as of now, there are 5 monitoring applications. Yes 5! Two of them are already imported in Qlik Sense after the installation and three others waiting to be manually imported in C:\Programdata\Qlik\Sense\Repository\DefaultApps

1111.png

Operation Monitor: Gives you a global overview of your environment’s healthiness. There you can find information about error and warning reported in the logs, task reloads, resource consumption, active users and more

License Monitor: Give you a global overview of your license usage helping you to manage license allocation more efficiently.

Log Monitor: An application going deep dive into your Qlik Sense logs to help you better understand what is happening in your environment.

Reloads Monitor: Will let you understand exactly how your applications are being reloaded, how often, how long, on which nodes and how successful those reloads are.

Session Monitor: Who is using your applications, when, where and how often. This application will also help you to understand which apps are popular or not in your environment.

How does the Reload work?

To display the relevant information, the Monitoring Applications are loading data from two data sources; The logs and the database.

As of Qlik Sense September 2017, it is now possible to store the logs into PostgreSQL database (Centralized logging) and/or files. If Centralized Logging has been configured in your environment, it will fetch the data from there. If it’s not configured or for some reason not accessible, it will load the data from the files.

Once loaded that information will be stored into a QVD file in C:\Programdata\Qlik\Sense\Log on the node doing the reload.

Each monitoring application is generating its own set of QVDs:

Operation Monitor: governance_date_time_ops_version.qvd, governance_time_range_ops_version.qvd

License Monitor: governanceLicenseLog_version.qvd and governanceSession_version.qvd

and governanceLogContent_version_file.qvd

Log Monitor: governanceLogMonitor_version.qvd

Reload Monitor: reloadsMonitorLogContent_version.qvd

Sessions Monitor: sessionsMonitorLogContent_version.qvd

 

In addition, for some of those QVD's you will see an extension _file or _db. and it depends from which source the monitoring applications are loading the data from. (Centralized logging or files)

22222.png

Finally, to load data from the database, the Monitoring Applications are using a set of Rest Connector data connections. Those are used to load different information such as users, applications, application objects, tasks, license, …

33333.png

 

Setup and configuration

Depending if you are running a single or multi-node environment and using Centralized Logging or not there are several steps to ensure the Monitoring Applications will fetch all the data in your environment.

I would suggest following the steps described here. Please make sure to select the correct version at the top right

44444.png

Once you have set up everything according to the link above depending on your environment, I’ll drive you through a couple of useful setting allowing you to customize your Monitoring Applications.

If you open the Script Load Editor for any of the Monitoring Applications (You will need to create an unpublish copy the application to do that)

In the first tab called Initialize, you will find two useful settings

SET db_v_file_override =             0; (line 10)

And

SET monthsOfHistory                     = 3; (line 24)

db_v_file_override: This variable defines which data source the application will use to loads its data. (Centralized Logging or Log files)

By default, this is set to 0 meaning that the application will check connectivity toward the Centralized Logging database and if it works it will load data from there. If it doesn’t then it will load from the log files.

By setting it to 1 you will force the application to load from log files and by setting it to 2 you will force it to load from the Centralized Logging.

monthsOfHistory: This variable control how much history will be available in the application. The default is 3 but you can increase/decrease it.

Keep in mind that more history available equal more data to load and process so it will increase time and resource consumption.

aaaaaaa.png

If you do any change in the script, don’t forget to publish the application again in the Monitoring Apps stream.

 

Finally, in a multi-node environment, by default, the Monitoring Apps will only be available on the Central Node.

This means that if you have a dedicated scheduler rim node or want to open those applications in the hub from a rim node, it won’t work.

This is control by the load balancing rule called ResourcesOnNonCentralNodes.

55555.png

Basically, this default rule says that every application will be accessible on every node, but the ones published in the Monitoring Apps stream.

To get rid of that you can, for instance, remove the highlighted text in the Conditions box which will make the Monitoring Applications available on every node or you can even custom the rule so that they are available on specific rim node(s) only.

 

Hope this will be clear and useful for hopefully many of you and don’t hesitate to like, comment and share (I will try to answer ALL your questions) 

 

 

 

 

10 Comments
Employee
Employee
This is terrific. Thank you.
0 Likes
2,486 Views
Or
Valued Contributor II

As of February 2019, it seems there's actually six monitoring apps shipping with QS: the five aforementioned ones and "Sense Performance Analyzer". No idea what it actually does, but it's in that same folder with the other five.

2,261 Views

Good point. I missed that one 🙂

So this new application collects the telemetry and usage data from your environment.
It also only runs with Qlik Logging Database enabled. It's currently a BETA version but I will run some test and update the blog post. Thank you for pointing this out! 

2,147 Views
chandnishah
New Contributor II

Does any of these apps capture sheet level usage?

1,507 Views
Employee
Employee

@chandnishah the operations monitor app that ships with Feb 19 includes a new sheet that captures sheet usage.

1,367 Views
Luminary
Luminary

Great to see the monitoring apps getting some love and attention!
The new icons are really nice too, btw.

Especially good with the configuration instructions - previously it's been somewhat of a trial and error process to re-configure the monitoring apps for various Sense environments - this article does improve that.

Some words of caution though.
This is how I have understood things. They are not ideal, so I'd love to be proven wrong...

If you have a multi-node Sense environment and allow for example the Operations Monitor to be reloaded on any node (or rather: on more than one non-central node), you should be aware of the following: 
The Operations Monitor (and I suspect the other apps too) store incremental QVDs in the Archived Logs data connection on the server where they are reloading. That is a directory on the server's local disk.

This means that if the Operations Monitor happens to reload on serverA the first time, but is sent to serverB the next time the scheduler task triggers, there will be no incremental QVDs on serverB. The entire reload will be re-done on serverB.
Let's say you have 4 reload servers. The Ops Monitor app will the over time be reloaded on all those servers, re-creating the same QVDs on the various servers.

This works, but seems kind of wasteful way of doing things. Better would be to store the QVDs on a central/shared data connection/folder - but that's on the other hand hard to set up in a general way out of the box. So it's a tricky one.

The solution could maybe be to provide some templates/guidance/instructions for what's needed to achieve this shared QVD storage?
I've done these changes myself, the bad part is that my customisations to the Operations Monitor app have to be manually copied to the the app version every time Sense is upgrade... That's a pain. 

Finally, a small feature request:
It would be useful with a flag that make it possible to always include full months in the Operations Monitor (and other apps). 
When comparing month-over-month metrics you need full months of data, otherwise the first month (which is partial) is useless. 

Thanks,
Göran 

1,156 Views
Partner
Partner

There could also be another new monitoring app called Connector Log Analyzer.
It was included into the insight preview April 2019, but I'm not sure it's into the final version.
(also, Sense Performance Analyzer should be out of BETA in April 2019 if I'm not wrong)

Riccardo

0 Likes
1,070 Views
oknotsen
Honored Contributor III

There could also be another new monitoring app called Connector Log Analyzer.
It was included into the insight preview April 2019, but I'm not sure it's into the final version.


The Connector Log Analyzer was already (accidentally) released with February 2019 SP2 (maybe even SP1).
It is released with April 2019 (again).

https://help.qlik.com/en-US/sense/April2019/Subsystems/Monitoring/Content/Sense_Monitoring/Connector...

 

1,057 Views

Hello @mountaindude,

Thank you for your comment! 

So regarding your first point, I'll try to explain how it works. 

The QVDs get stored and loaded in the data connection called "ServerLogFolder" which by default points to "C:\Programdata\Qlik\Sense\Log". So for a default single node environment, everything is fine. 

Now coming to the multinode setup, as per the instructions provided here you will need to change the value of the "ServerLogFolder" data connection to point to \\CentralNodeServer\Sense\Log\. 

You will also need to create one "ServerLogFolder" per RIM node and add these connection string in the monitoring applications script such as (See the provided link for more detailed instructions)

  • ServerLogFolderRIM1 pointing to \\RIM1\Sense\Log\
  • ServerLogFolderRIM2 pointing to \\RIM2\Sense\Log\
  • ServerLogFolderRIM3 pointing to \\RIM3\Sense\Log\
  • ...

But since you changed the data connection called "ServerLogFolder" to point to a shared folder, now the QVDs will only be stored and read from the shared path only even though you are running the reload on a rim node. 

Hope that makes sense and let me know if you have any question.

Regarding your second point with the flag filtering on "Completed Month" only, I will log this as a feature request.

Thank you again for your comment and hope my answer helped!

0 Likes
979 Views
Luminary
Luminary
Ahhh, I totally missed that documentation about multi-node setup.
Then all is fine - things are properly documented! 🙂

Thanks!

844 Views