Qlik Analytics: How to log CPU, Disk, and memory usage with Microsoft Performance Monitor on a windows 2012 Server (PerfMon)

[Andre_Sostizzo]

The Microsoft Performance Monitor makes it possible for users to set up automatic data collection jobs that can save data from performance counters at a given interval. 

This document describes the steps necessary to save such data using Microsoft Performance Monitor Data Collectors. 

Environment

• QlikView Server, all versions
• Qlik Sense
   

Prerequisites

• Microsoft Performance Monitor must be installed on the QlikView or Qlik Sense Server
• The user who initiates the Data Collectors must be at least a local admin

Notes

• These include the Qlik Sense-specific and some other environmental counters, but not all counters that may apply to different troubleshooting scenarios. (e.g. Networking, disk read-write frequency and speed, etc.)
• Attached to this article, you can find an XML template including the counters below with additional disk and network counters. The template file can be imported in Performance Monitor under Data Collector Sets > right-click User Defined > New > Data Collector Set > Create from a template.
• Also see the following article, which contains an illustration of the steps below when adding a processor time counter for the QVS process. How to set up performance monitoring for QlikView Server Service(QVS) (perfmon)  

Setting up a Data Collector

1. Open Microsoft Performance Monitor (located under Server Manager->Tools)
2. Open "Data Collector Sets" in the tree view on the left pane
3. Open "User Defined"
4. Right-click on "User Defined" and select "New" -> "Data Collector Set"
5. Set a name for the new collector and select the option "Create manually (Advanced)". Click "Next"
6. Select "Create data logs" and check "Performance counter". Click "Next"
7. Click "Add..."
8. Expand "Processor" and select "% Processor Time". Make sure that "_Total" are selected in "Instances of selected object:"
   1. Expand “Memory” and select “Available Mbytes”, and "Committed Bytes"
   2. Expand "Logical Disk" and select "Free Megabytes"
   3. Expand "Physical Disk" and select "% Idle Time", "Avg.  Disk sec/Read"
9. Click "Add >>"
10. Expand "QlikView Distribution Service" and select every sub-item available
11. Click "Add >>"
12. Click "Ok" to close the dialog
13. Change the "Sample Interval" to "5". Make sure that "Units" reads "Seconds"
14. Click "Next"
15. Choose a directory to save the logs in. Click "Next"
16. Make sure that the Data Collector Set are run as a user with Local Admin credentials. Select "Save and Close" and click "Finish"
17. Select the newly created Data Collector Set in the tree view on the left pane
18. Right-click "DataCollector01" in the right pane and select "Properties"
19. Change "Log Format" to "Comma Separated"
20. Open the "File"-tab and enter a descriptive name for the log file, preferably Servername. In a Publisher cluster, this is mandatory so the data can be related to which cluster node it comes from. 
21. Click "OK"
22. Right-click the Data Collector Set in the tree view on the left pane and select "Start"
23. Wait a few seconds and verify that the Data Collector Set is running
24. Repeat for each cluster node

Disk Latency Counters:

While there are no hard values when it comes to transfer speed for storage or latency, we want to minimize I/O request queues as this leads to high disk latency. So a high Disk Queue Length leads to high Disk sec/Read and Disk sec/Write times, and consequently to a higher Disk sec/Transfer which is the actual measure of disk response time/latency.

 So, when it comes to which performance counters to set for storage monitoring you should use:

\\PhysicalDisk\Current Disk Queue Length

\\PhysicalDisk\Avg. Disk Queue Length

\\PhysicalDisk\Avg. Disks Read Queue Length

\\PhysicalDisk\Avg. Disk Write Queue Length

\\PhysicalDisk\Avg. Disk sec/Transfer

\\PhysicalDisk\Avg. Disk sec/Read

\\PhysicalDisk\Avg. Disk sec/Write)

Then pick again the same fields for LogicalDisk. 

More information on what may be considered healthy or unhealthy disk response times  can be found via links below:

• Measuring Disk Latency with Windows Performance Monitor (Perfmon) 
• Windows Performance Monitor Disk Counters Explained