With February 2018 of Qlik Sense, it is possible to capture granular usage metrics from the Qlik in-memory engine based on configurable thresholds. This provides the ability to capture CPU and RAM utilization of individual chart objects, CPU and RAM utilization of reload tasks, and more.
Also see Telemetry logging for Qlik Sense Administrators in Qlik's Help site.
Click here for Video Transcript
Enable Telemetry Logging
In the Qlik Sense Management Console, navigate to Engines > choose an engine > Logging > QIX Performance log level. Choose a value:
- Off: No logging will occur
- Error: Activity meeting the ‘error’ threshold will be logged
- Warning: Activity meeting the ‘error’ and ‘warning’ thresholds will be logged
- Info: All activity will be logged
Note that log levels Fatal and Debug are not applicable in this scenario.
Also note that the Info log level should be used only during troubleshooting as it can produce very large log files. It is recommended during normal operations to use the Error or Warning settings.
- Repeat for each engine for which telemetry should be enabled.
Set Threshold Parameters
- Edit C:\ProgramData\Qlik\Sense\Engine\Settings.ini If the file does not exist, create it. You may need to open the file as an administrator to make changes.
- Set the values below. It is recommended to start these threshold values and increase or decrease them as you become more aware of how your particular environment performs. Too low of values will create very large log files.
[Settings 7]
ErrorPeakMemory=2147483648
WarningPeakMemory=1073741824
ErrorProcessTimeMs=60000
WarningProcessTimeMs=30000
- Save and close the file.
- Restart the Qlik Sense Engine Service Windows service.
- Repeat for each engine for which telemetry should be enabled.
Parameter Descriptions
- ErrorPeakMemory: Default 2147483648 bytes (2 Gb). If an engine operation requires more than this value of Peak Memory, a record is logged with log level ‘error’. Peak Memory is the maximum, transient amount of RAM an operation uses.
- WarningPeakMemory: Default 1073741824 bytes (1 Gb). If an engine operation requires more than this value of Peak Memory, a record is logged with log level ‘warning’. Peak Memory is the maximum, transient amount of RAM an operation uses.
- ErrorProcessTimeMs: Default 60000 millisecond (60 seconds). If an engine operation requires more than this value of process time, a record is logged with log level ‘error’. Process Time is the end-to-end clock time of a request.
- WarningProcessTimeMs: Default 30000 millisecond (30 seconds). If an engine operation requires more than this value of process time, a record is logged with log level ‘warning’. Process Time is the end-to-end clock time of a request.
Note that it is possible to track only process time or peak memory. It is not required to track both metrics. However, if you set ErrorPeakMemory, you must set WarningPeakMemory. If you set ErrorProcessTimeMs, you must set WarningProcessTimeMs.
Reading the logs
- To familiarize yourself with logging basics, please see help.qlik.com
Note: Currently telemetry is only written to log files. It does not yet leverage the centralized logging to database capabilities.
- Telemetry data is logged to C:\ProgramData\Qlik\Sense\Log\Engine\Trace\<hostname>_QixPerformance_Engine.txt
- and rolls to the ArchiveLog folder in your ServiceCluster share.
- In addition to the common fields found described), fields relevent to telemetry are:
- Level: The logging level threshold the engine operation met.
- ActiveUserId: The User ID of the user performing the operation.
- Method: The engine operation itself. See Important Engine Operations below for more.
- DocId: The ID of the Qlik application.
- ObjectId: For chart objects, the Object ID of chart object.
- PeakRAM: The maximum RAM an engine operation used.
- NetRAM: The net RAM an engine operation used. For hypercubes that support a chart object, the Net RAM is often lower than Peak RAM as temporary RAM can be used to perform set analysis, intermediate aggregations, and other calculations.
- ProcessTime: The end-to-end clock time for a request including internal engine operations to return the result.
- WorkTime: Effectively the same as ProcessTime excluding internal engine operations to return the result. Will report very slightly shorter time than ProcessTime.
- TraverseTime: Time spent running the inference engine (i.e, the green, white, and grey).
Note: for more info on the common fields found in the logs please see help.qlik.com.
Important Engine Operations
The Method column details each engine operation and are too numerous to completely detail. The most relevent methods to investigate are as follows and will be the most common methods that show up in the logs if a Warning or Error log entry is written.
| Method |
Description |
| Global::OpenApp |
Opening an application |
| Doc::DoReload, Doc::DoReloadEx |
Reloading an application |
| Doc::DoSave |
Saving an application |
| GenericObject::GetLayout |
Calculating a hypercube (i.e., chart object) |
Comments
- For best overall representation of the time it takes for an operation to complete, use ProcessTime.
- About ERROR and WARNING log level designations: These designations were used because it conveniently fit into the existing logging and QMC frameworks. A row of telemetry information written out as an error or warning does not at all mean the engine had a warning or error condition that should require investigation or remedy unless you are interested in optimizing performance. It is simply a means of reporting on the thresholds set within the engine settings.ini file and it provides a means to log relevant information without generating overly verbose log files.
Qlik Telemetry Dashboard:
Once the logs mentioned above are created, the Telemetry Dashboard for Qlik Sense can be downloaded and installed to read the log files and analyze the information.
The Telemetry Dashboard provides the ability to capture CPU and RAM utilization of individual chart objects, CPU and RAM utilization of reload tasks, and more.
For additional information including installation and demo videos see Telemetry Dashboard - Admin Playbook
The dashboard installer can be downloaded at: https://github.com/eapowertools/qs-telemetry-dashboard/wiki. NOTE: The dashboard itself is not supported by Qlik Support directly, see the Frequently Asked Questions about the dashboard, to report issues with the dashboard, do so on the "issues" webpage.
1. Right-click installer file and use "Run as Administrator". The files will be installed at C:\Program Files\Qlik\Sense
2. Once installed, you will see 2 new tasks, 2 data connections and 1 new app in the QMC.
3. In QMC Change the ownership of the application to yourself, or the user you want to open the app with.
4. Click on the 'Tasks' section in the QMC, click once on 'TelemetryDashboard-1-Generate-Metadata', then click 'Start' at the bottom. This task will run, and automatically reload the app upon completion.
5. Use the application from the hub to browse the information by sheets
