How to Start Investigating a Suspected Qlik Sense Performance Issue

[Sue_Macaluso]

[Video embedded here. Use Insert Video button in the menu bar]

Transcript

Hello everyone!
This video aims at providing initial guidance when it is suspected that Qlik Sense may be experiencing a performance issue.We will refer to performance from an end-user point of view. This means it relates to the Qlik Sense server response time for requests made by the client.
As a fist example, we will look at a performance issue reported with only a specific App.
Determining whether a group of apps experience the same performance issue is useful as it helps to determine development and design patterns and draw further conclusions.
In this report however, only one specific App is known to have never had an acceptable response time when opening and rendering objects.
So we do not seek to correlate a specific past development or design modification in the App as the cause, nor a possible changes to the source data or ETL process as being the reason for the App's poor response time.
One commonly found cause for response time delays is that the App data model may need to be optimized.
The Qlik Sense "App Metadata Analyzer" Monitoring App may help gain insight into data model problems and app footprint considerations.
Here we see an example of identified synthetic keys and data islands which may cause different problems.
More information on the App Metadata Analyzer can be found in Qlik's Help site on this page. (https://help.qlik.com/en-US/sense-admin/September2020/Subsystems/DeployAdministerQSE/Content/Sense_D...)
However, note that data model problems are not limited to unintended synthetic keys and data islands, which are generally not recommended to have in Apps. (https://help.qlik.com/en-US/sense/September2020/Subsystems/Hub/Content/Sense_Hub/Scripting/synthetic...)
It is also recommended that the single-threaded operations related to object calculations be optimized.
Here we see that after opening an app and rendering a problematic object, there are spikes only at specific CPUs at different times.
This means that this object is taking longer to process a single-threaded operation.
For more information see the knowledge base article "Single and Multi-threading in QlikView and Qlik Sense". (https://community.qlik.com/t5/Support-Knowledge-Base/Single-Multi-threading-in-QlikView/ta-p/1716291)
Note that in cases where a specific App has performed better in the past, the reported time to open the app should be compared to what has previously been recorded as a baseline for the App.
This means investigating possible changes in the App's dev. and design, source data, and ETL setup that may play a role in the increased response time.
In cases like this a previously obtained baseline for comparison can be useful. Administrators may use the Qlik Scalability Tools in order to determine performance baselines for the Qlik Sense environment and specific Apps. See the Qlik Sense Scalability Tools information in Community. (https://community.qlik.com/t5/Support-Knowledge-Base/Qlik-Sense-Scalability-Tools/ta-p/1715051)
For investigating potential App behavior anomalies and when it may have started, another useful tool called the Telemetry Dashboard may be used.
This tool relies on Qix Performance logs which need to be enabled at Warning level. For more information see "How to enable Qlik Sense QIX performance logging and use the Telemetry Dashboard" in Qlik Community (https://community.qlik.com/t5/Support-Knowledge-Base/How-to-enable-Qlik-Sense-QIX-performance-loggin...), and Help site. https://help.qlik.com/en-US/sense-admin/September2020/Subsystems/DeployAdministerQSE/Content/Sense_D....
Another common performance issue example is when users report that all Qlik Sense Apps, including newly created simple apps take a long time to open and render objects.
The report is that the problem has always occurred at any given time of the day, specially when users need the Apps the most.
So the issue does not seem related to a scheduled local event but should verified by an administrator.
Local events that should be confirmed not to happen while the issue occurs include but are not limited to a period coinciding with scheduled App reload task runs, scheduled service re-starts, antivirus full system scans, server backup procedures, or others.
One suitable next step is to confirm whether the reported issue still occurs when the App is opened while logged into the Qlik Sense server itself.
Or at least on another machine with very close network proximity to the Qlik Sense server. Say another machine on the same LAN segment, preferably on the same switch.
If the issue does not occur locally, then a reasonable next step would be to investigate if the problem may be related to network latency or a client related environmental problem.
Otherwise, testing with a different Qlik Sense user such as an administrator with less environmental restrictions on different client computers in order pinpoint patterns would also be applicable.
This also includes testing with an account that has Qlik Sense security rules that are highly permissive, or that are at least very generalized, with a high score.
For more information on how this may apply to the perceived performance see the Support post "Security Rules and Performance in Qlik Sense (https://community.qlik.com/t5/Support-Updates-Blog/Security-Rules-and-Performance-in-Qlik-Sense/ba-p...)
Keep in mind also that using Section Access may affect the perceived performance depending on how it is implemented. See article "Delay on opening App when section access is used". (https://community.qlik.com/t5/Support-Knowledge-Base/Delay-on-opening-App-when-section-access-is-use...)
Now if the problem occurs regardless of from where the Qlik Sense Apps are being accessed from and which Qlik Sense user is accessing them, this may warrants further investigation on the Qlik Sense server itself.
Looking at the Engine Trace logs is useful in determining if the server may not have enough memory or CPU resources to process all requests within the expected time-frame.
In this example, the logs show that the Qlik Sense Engine low and high working set memory limits have been reached and grown beyond the specified parameters.
The "working set" parameters are found in the Engine's Advanced settings, as the memory usage here pertains to the Engine service.
In general, if there is a warning indicating that the upper Engine RAM usage limit was reached in the logs, and the limit is set to the default values of 90%, then the system may need more RAM.
However, this may depend on how much RAM the system actually still has available. For more information on working set thresholds see the following knowledge base article in Qlik Community. (https://community.qlik.com/t5/Support-Knowledge-Base/What-does-quot-Warning-WorkingSet-Virtual-Memor...)
We can also look for the Windows Event ID 2004 registered in the system event logs. This may help confirming that the system is truly out of available RAM.
More information is available in this community article. (https://community.qlik.com/t5/Support-Knowledge-Base/Windows-successfully-diagnosed-a-low-virtual-me...)
Note however that it is important to determine if the Engine's high memory usage is not being caused by concurrent App reload tasks that could be decreased or run at off-peak usage hours.
More information on App reload concurrency is available under this article. (https://community.qlik.com/t5/Support-Knowledge-Base/Concurrent-Reload-Settings-in-Qlik-Sense-Enterp...)
It is also important to confirm if there are highly consumed App(s) that could be further optimized.
Information on obtaining App memory use and how it relates to the server overall performance can be found under the article "How much RAM is consumed per application?" (https://community.qlik.com/t5/Support-Knowledge-Base/How-much-RAM-is-consumed-per-application/ta-p/1...)
In some scenarios however, latency in server response time may be experienced when not observing high memory utilization, but with persistent high CPU usage on all cores.
Generally, if the average CPU utilization is around 70% or higher, the system may not have enough CPU power and may be oversubscribed. For more information see Qlik Supports documentation on "QIX engine memory management and CPU utilization". (https://community.qlik.com/t5/Support-Knowledge-Base/QIX-engine-memory-management-and-CPU-utilizatio...)
Another useful tool is the Qlik Sense System Performance Analyzer monitoring App, which may also help drawing further RAM and CPU usage conclusions. Find more information under the Qlik Help site. (https://help.qlik.com/en-US/sense-admin/September2020/Subsystems/DeployAdministerQSE/Content/Sense_D...)
In systems struggling with slow response times, high CPU, or RAM usage, it is common to also find the following AAALR warnings registered in the same Engine trace logs mentioned prior.
For more information on steps that may help improve performance see the following article in Qlik Community. (https://community.qlik.com/t5/Support-Knowledge-Base/AAALR-is-registered-in-the-Qlik-Sense-logs/ta-p...)
Otherwise, if not due to high RAM or CPU usage, the latency may be due to some other environmental factors.
One common factor is that Antivirus software may slow down requests if Qlik Sense operational files are not properly excluded from real-time scanning.
For more information please see the following Knowledge Base article on Qlik Community which also contains the link to related documentation in the Help site. (https://community.qlik.com/t5/Support-Knowledge-Base/Antivirus-exceptions-for-Qlik-Sense-McAfee-Syma...)
Other environmental factors to consider include but is no limited to the following:
Disk read and write speed, share persistence and QRS database connection latency, and server hardware characteristics and configuration;
For information on Share Storage options and requirements, see the following article. (https://community.qlik.com/t5/Support-Knowledge-Base/Qlik-Sense-Share-Storage-options-and-requiremen...)
Persistence requirement for the repository database can be found under the following page in Qlik's Help site. (https://help.qlik.com/en-US/sense-admin/September2020/Subsystems/DeployAdministerQSE/Content/Sense_D...)
Windows Performance Counters may help pinpointing performance bottlenecks beyond CPU and RAM constrains. See the knowledge base article on "How to monitor resources using Microsoft Performance Monitor". (https://community.qlik.com/t5/Support-Knowledge-Base/Qlik-Sense-How-to-monitor-resources-using-Micro...)
Note also that different server hardware configurations may lead to different overall performance behavior.
There are BIOS and OS settings that are known to improve performance. See the knowledge base article "Hardware settings for best performance with Qlik Sense" for more information. (https://community.qlik.com/t5/Support-Knowledge-Base/Hardware-Settings-For-Best-Performance-With-Qli...)
There are also known servers that have been validated to have favorable processors.
See the "List of Favorable Server Hardware from Qlik Scalability Center" knowledge base article for more information in Qlik Community. (https://community.qlik.com/t5/Support-Knowledge-Base/List-of-Favorable-Server-Hardware-from-Qlik-Sca...)
The previously mentioned Scalability Tools includes the Hardware Benchmarking Package. (https://community.qlik.com/t5/Support-Knowledge-Base/Qlik-Sense-Scalability-Tools/ta-p/1715051)
However, for general guidance prior Qlik Sense deployment, please review the latest Qlik Performance Benchmark which can be obtained via the following article. (https://community.qlik.com/t5/Support-Knowledge-Base/Qlik-Sense-Performance-Benchmark/ta-p/1712120)
Lastly, when using virtualization, please review the article "Virtualization Best Practices In QlikView And Qlik Sense" (https://community.qlik.com/t5/Support-Knowledge-Base/Virtualization-Best-Practices-In-QlikView-And-Q...)
For additional in-depth assistance with performance related inquiries, Qlik provides consulting services. Please see the Knowledge Base article "How and When to Contact the Consulting Team?" (https://community.qlik.com/t5/Support-Knowledge-Base/How-and-When-to-Contact-the-Consulting-Team/ta-...)
If you’d like more information,
search for answers using the unified search tool on the Support Portal.
It searches across the support knowledge base, Qlik Community, Qlik Help site, and Qlik YouTube channels.
Take advantage of the expertise of peers, product experts, and technical support engineers by asking a question in a Qlik Product Forum on Qlik Community.
And don’t forget to subscribe to the Support Updates Blog.
Thanks for watching!