Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.
Search our knowledge base, curated by global Support, for answers ranging from account questions to troubleshooting error messages.
After upgrading Qlik Enterprise Manager to 2022.11, increased connection errors to the Qlik Replicate server can be observed, with the connection terminating approximately every 30 minutes.
The following error is logged:
3 2023-12-19 06:09:06 [ServerDto ] [ERROR] Failed to get monitoring data for server: 172.29.73.29. Message:'SYS-E-SESS_LOST, The client is not connected to the Server. Please connect and then try again.'''.
This is a known defect. The fix will be included in the next possible Qlik Enterprise Manager release (tentatively 2023.11 SP03).
Fix Description: Added auto-reconnect flag where it was missing to allow QEM to better manage the connection drops and attempt to re-establish the EM<-->QR communication, especially in high-load situations.
The following are test patches that address this issue. Please contact support to obtain the test patch.
Qlik Enterprise Manager 2022.11.0.634
Qlik Sense Enterprise on Windows
Result files cannot be found. Depending on which tool is used (Exerciser, Gopherciser or JMeter), the extension and format is different. At the time of writing this article (March 2024), the latest version of Qlik Sense Enterprise scalability tools is using Gopherciser by default.
Set the extension for the file in the "Main" tab (either .log or .tsv)
In the folder structure linked to the data connection "DataFiles", make sure that the result file has the correct extension (.tsv in this example) and is placed in the Gopherciser folder.
The TimeSlot field is created based on the time field (case-sensitive), make sure this column is included in the result file.
//Tab: __SUB LoadGopherciser
Load
'Gopherciser' as Origin,
FileDir() as path,
Timestamp(Timestamp#(Timestamp(Mid(Replace("time",'T', ' '),1,16),'YYYY-MM-DD hh:mm') & ':00', 'YYYY-MM-DD hh:mm:ss')) as TimeSlot, // Local time
"time" as Org_TimeStamp,
...
Scalability Results Analyzer
Qlik Sense Enterprise Scalability Tools
A Qlik Replicate task can generally be stopped from the Qlik Replicate web console. However, sometimes stopping the task takes longer than anticipated. A delay in the task being stopped can be caused by, for example:
If you plan to reload the task or to start it from a timestamp, you have the option to kill or terminate the task at the operating system level, as the task will not need to reference its stop position to resume where it left off. A reload or start from a timestamp is considered a fresh start of the task.
Never manually terminate or kill the task if you intend to resume the task and you do not see this entry in the task log:
00009760: 2022-05-23T23:30:49 [UTILITIES ]I: The last state is saved to file 'D:\Program Files\Attunity\Replicate\data\tasks\TASK_NAME/StateManager/ars_saved_state_000002.sts' at Tue, 24 May 2022 04:30:41 GMT (1653366641301364) (statemanager.c:642)
After the task has been terminated, please check with your source and target DBA to ensure any orphaned connections are ended.
Qlik Replicate, all versions
Windows
The information in this article is provided as-is and to be used at own discretion. Depending on tool(s) used, customization(s), and/or other factors ongoing support on the solution below may not be provided by Qlik Support.
After upgrading to Qlik Sense Enterprise on Windows May 2023 (or later), the Qlik Sense Repository Service may cause CPU usage spikes on the central node. In addition, the central Engine node may show an increased average RAM consumption while a high volume of reloads is being executed.
The Qlik Sense System_Repository log file will read:
"API call to Engine service was successful: The retrieved 'StaticByteSize' value for app 'App-GUID' is: 'Size in bytes' bytes."
Default location of log: C:\ProgramData\Qlik\Sense\Log\Repository\Trace\[Server_Name]_System_Repository.txt
This activity is associated with the ability to see the Base memory size in the Qlik Sense Enterprise Management Console. See Show base memory size of apps in QMC.
The feature to see Base memory size can be disabled. This may require an upgrade and will require downtime as configuration files need to be changed.
Take a backup of your environment before proceeding.
If any issues occur, revert the changes by restoring the backed up files and open a ticket with Support providing the changed versions of repository.exe.config, capabilities.json mentioning this article.
Show base memory size of apps in QMC
QB-22795
QB-24301
Qlik Sense Enterprise on Windows May 2023, August 2023, November 2023, February 2024.
It is finally here: The first public iteration of the Log Analysis app. Built with love by Customer First and Support.
"With great power comes great responsibility."
Before you get started, a few notes from the author(s):
Chapters:
01:23 - Log Collector
02:28 - Qlik Sense Services
04:17 - How to load data into the app
05:42 - Troubleshooting poor response times
08:03 - Repository Service Log Level
08:35 - Transactions sheet
12:44 - Troubleshooting Engine crashes
14:00 - Engine Log Level
14:47 - QIX Performance sheets
17:50 - General Log Investigation
20:28 - Where to download the app
20:58 - Q&A: Can you see a log message timeline?
21:38 - Q&A: Is this app supported?
21:51 - Q&A: What apps are there for Cloud?
22:25 - Q&A: Are logs collected from all nodes?
22:45 - Q&A: Where is the latest version?
23:12 - Q&A: Are there NPrinting templates?
23:40 - Q&A: Where to download Qlik Sense Desktop?
24:20 - Q&A: Are log from Archived folder collected?
25:53 - Q&A: User app activity logging?
26:07 - Q&A: How to lower log file size?
26:42 - Q&A: How does the QRS communicate?
28:14 - Q&A: Can this identify a problem chart?
28:52 - Q&A: Will this app be in-product?
29:28 - Q&A: Do you have to use Desktop?
Qlik Sense Enterprise on Windows (all modern versions post-Nov 2019)
*It is best used in an isolated environment or via Qlik Sense Desktop. It can be very RAM and CPU intensive.
The information in this article is provided as-is and to be used at own discretion. Depending on tool(s) used, customization(s), and/or other factors ongoing support on the solution below may not be provided by Qlik Support.
Optimizing Performance for Qlik Sense Enterprise - Qlik Community - 1858594
QlikView and Qlik Sense files are like ZIP file, as in they are highly compressed. So obtaining an estimate of how much RAM is needed for loading Apps into memory (footprint) is difficult to provide across the board. The expected memory usage associated with the app can also depend on how many users access the app and how much they are used.
An estimate of the RAM needed per app can be built on the below, but for accuracy always test by loading the app into memory and using the Qlik Scalability Tools to obtain a baseline of memory usage for each app as it is accessed by the foreseeable number of users.
Telemetry logging which allows for using the Telemetry Dashboard, is also an available suitable tool.
Since the release of the February 2019 version, Sense System Performance Analyzer monitoring App can be used to determine app footprint as well. Since the release of the June 2018 version, App Metadata Analyzer monitoring App can also be used. However, these last three may not provide the same data as the Scalability Tools.
An alternative is to use a test environment and simply observe RAM usage increases as apps are first opened (footprint), and as number of users accessing the app increase, then as number of operations are performed within the app. The test environment results for observed memory and CPU usage can be recorded as a baseline for the particular app in production, which assists in determining future sizing/scaling needs.
Need direct assistance in evaluating your Qlik Sense and QlikView apps? Qlik's Professional Services are available to assist you.
RAMInitial = SizeOnDisk × FileSizeMultiplier ; this is the initial RAM footprint for any application
FileSizeMultiplier: range between 2-10 (this is a compression* ratio depending on the data contained in the app)
*Compression is based upon the data, and how much we can compress depends upon the homogeneity of the data. The more homogeneous, the more compression Qlik can achieve.
More information on one way of optimizing a document can be found on the Qlik Design Blog: Symbol Tables and Bit-Stuffed Pointers
RAMperUser = RAMinitial × userRAMratio ; this is the RAM per each incremental user
userRAMratio: range between 1% -10%
Total RAM used per app :
TotalRAM = (RAMperUser × Number of users) + RAMinitial
Example:
SizeOnDisk = 1 GB
File Size Multiplier 6 (range is from 2 to 10) This is an example value. A value can be obtained by reviewing telemetry (how far does the app expand?) or opening the app in a QlikView or Qlik Sense desktop and measuring the expansion based on how large it grows in memory.
RAMinitial = 1 * 6 = 6 GB
If we take a RAM Ratio per User of 6% then it is 0.06 (range is from 1 to 10)
RAMperUser = 6 x 0.06 = 0.36GB
Then RAM Required for 30 Users
TotalRAM = (RAMperUser × No. users) + RAMinitial
TotalRAM = (0.36 * 30) + 6
TotalRAM = 16.8 GB
NUMA, Non-Uniform Memory Access, is a computer memory design used in Multiprocessing, where the memory access time depends on the memory location relative to a processor. Under NUMA, a processor can access its own local memory faster than non-local memory, that is, memory local to another processor or memory shared between processors.
QlikView generally performs better with NUMA disabled. The NUMA setting is controlled by the computer BIOS, and the recommendation is to disable it for best possible performance.
Some hardware does not allow disabling NUMA. For these scenarios QlikView 11.20 SR2 and later can automatically detect the presence of NUMA and if present adjust the QVS.exe to adapt to a NUMA environment. While not as efficient as disabling NUMA in BIOS, it restores some of the performance lost when NUMA is enabled.
The automatic detection only works in 2 or 4-socket systems. For 8 socket machines, the NUMA detection fails, leaving the QVS.exe to believe itself to exist in a non-NUMA environment.
The software NUMA adaption is controlled with the EnableNumaRoundRobin variable in the QlikView Server settings.ini file.
All recommendations can be found on the community: Quick tips #8 - Server settings for best performance they are being updated if Performance & Scalability team sees that changes in server configurations impact the performance. So it worth checking them from time to time.
Please keep in mind that the Performance & Scalability team tries to test many hardware configurations, but it is still a very small subset of possible server configurations seen in customer deployments. Due to this, in case of doubt, they suggest testing the impact of the setting. For example, one customer might on average see that for their deployment disabling NUMA in BIOS is beneficial, while others might not notice any difference.
All changes should be modified with caution, because set NUMA (in hardware and software) improperly might result in (significant) performance degrade, or fatal memory issue in some rare case.
EnableNumaRoundRobin is a setting configurable in the settings.ini file.
0 (default): Automated state where QlikView Server at startup detects whether NUMA is enabled or disabled at certain hardware and adapts to it. If NUMA is enabled QlikView Server will ignore NUMA nodes when allocating memory.
1: QlikView Server is acting as if NUMA is enabled from BIOS (without any attempts to detect it)
2: QlikView Server will not ignore NUMA nodes if present (i.e. NUMA enabled)
Modify the settings.ini and add EnableNumaRoundRobin=X, where X is your desired value. The settings.ini file is (by default) stored in %ProgramData%\QlikTech\ on your QlikView server node.
For information on how to change the settings.ini, see How to Modify the QlikView Server Settings.ini file.
A similar method can be used in the QlikView Desktop client. The settings used are slightly different.
EnableNumaRoundRobin (integer type parameter) for QlikView Desktop:
0 (default value): detect NUMA automatically and enable it in QlikView if possible
1: force QlikView Desktop to enable NUMA
Other: force QlikView Desktop to disable NUMA
For details of instruction and different testing result please refer to attached pdf file.
Qlik Replicate SAP Extractor Endpoint can be tuned to improve the performance of data being processed from an SAP environment. This is achieved by adding an internal parameter: sqlConnector. To confirm the performance improvements of the task, first record the task duration before implementing the parameter, then compare it to the subsequent runs.
Before the internal parameter can be added to the Qlik Replicate Extractor Endpoint, the below SAP Transports must be installed:
Once the transports have been installed make sure you follow the below steps to set the added Permissions on the SAP Replicate User:
Add the following authorization objects and settings to the RFC user ID:
Do the following:
For more information on how to import/upgrade the sqlConnector Transport, see Importing the SAP transports.
Configure the SAP Extractor Endpoint by defining the useSqlConnector Internal Parameter.
SAP Transports
Importing the SAP transports
After an upgrade to Qlik Sense Enterprise on Windows February 2023 or later, reloads may take longer to complete.
Example:
A Qlik Sense test script app (CTRL+00) may have an increased reload time of two minutes, rather than the expected two to ten seconds.
The issue was identified and mitigated in collaboration with Merck Group and Cognizant (Thiago Cunha, Akash Pohare, Vincent Wei), alongside Qlik's internal development and support teams (Vijay Vekariya, Oystein Kolsrud, Sebastian Linser, Eugene Sleator).
Disable SaveScriptWithSetScript. This disabled any additional save script calls and speeds up the reload.
To do so:
[Settings 7]
SaveScriptWithSetScript=0
In February 2023, Qlik introduced a collaborative feature allowing co-development in load scripts. This feature is intended for Qlik Cloud only. See Collaborate on load scripts in shared spaces. It can be safely disabled in an on-premise installation.
As of the February 2024 release, the feature will be disabled by default.
QB-23619
QB-22810
Qlik Sense Enterprise on Windows February 2023 - November 2023
Qlik Catalog Data loads fail to complete due to an out of memory error.
The issue can be addressed by either increasing memory on the host running Qlik Catalog and engine or by decreasing the number of concurrent loads Catalog will process (32 by default). You may need to do both.
Decrease the number of concurrent loads to 4.
The setting is changed in the core_env.properties file located in the directory: QDC_HOME/<Tomcat>/conf/core_env.properties.
A Tomcat restart is required after making the change in the core_env.properties file.
Make the change when no loads are running. Find and edit the following (do not add a new entry):
# Number of Java threads in pool used for each of loading data, prepare and publish.
# Restart required. Default: 32
hadoop.job.poolsize=4
If the loads are successful, increase the count gradually and continue testing.
If it is easy to provision more RAM, consider doing so.
Qlik Catalog uses the "engine" container to convert QVDs to CSVs (in order to sample and profile the data). The issue is that engine is running out of memory attempting to concurrently process several QVDs. Sometimes engine gracefully handles the memory exhaustion and returns an error to Qlik Catalog. Other times engine crashes and the container is restarted. In both cases, the job status is FAILED in Qlik Catalog.
If you experience delayed or stuck automations in the Blendr.io platform, please note that we are evaluating measures to address these issues, such as code updates and increased capacity. However, due to the previously announced end of life of the product, a long-term solution may not be possible.
When you encounter a stalled automation, you can still open a support case. Support will then manually set the automation to failed in order to allow any queued automations to continue.
Further, for troubleshooting purposes, please note the following:
You have been running Qlik Sense normally for quite some time. Over time you have accumulated an abundance of reload tasks configured in your Qlik Sense QMC.
Recently however, you have noticed that Qlik Sense QMC reload tasks are in the following state:
To resolve the issue, it is recommended to add an additional scheduler node (or nodes) in order to manage the ever increasing number of reload tasks in the affected Qlik Sense environment.
As time goes on, add additional scheduler nodes in proportion to the increasing number of reload tasks added/deployed in your environment.
The existing Qlik Sense scheduler nodes simply cannot manage the additional burden placed upon it by ever increasing reload tasks.
Concurrent Reload Settings in Qlik Sense Enterprise
QB-20013
If Oracle is configured with a Physical Standby, and the Oracle LOG_ARCHIVE_DEST_n initialization parameter is configured with the DELAY option, running a Full Load and Apply Changes task on an Oracle Physical Standby instance may result in missing or erroneous data in the target database.
The following warning may be thrown in the log files:
Using standby database could have caused the missing events problem during Full Load and CDC synchronization.
If you do not have the DELAY parameter configured and there is no data loss, this warning message can be ignored. Please check with your DBA if the DELAY parameter is set.
This issue can be resolved using an internal parameter: standbyDelayTime, which should be set to a value little bit bigger than the Delay parameter of LOG_ARCHIVE_DEST_n initialization parameter. Units are in minutes.
For information on how to set internal parameters, see Qlik Replicate: How to set Internal Parameters and what are they for?
This solution may lead to redundant records for a table without PK/UI if there were INSERT operations during that period. Example - LOG_ARCHIVE_DEST_2 = 'DELAY=10'
When installing and using the DB2 z/OS endpoint in Qlik Replicate, WLM tasks on the mainframe may get cancelled. There are multiple root causes for this, but a common one can be addressed by reviewing the execution priority (WLM settings).
The Full Load processes will utilize any available ZIIP processors. The best practice is to set the performance goal so that the processes get classified as lower than online workloads. In a very busy system, you might consider setting Honor Priority to NO for the service class assigned to the Full Load. This requires z/OS 2.1 + OA50845 or z/OS 2.2 and above.
Best practice is to set the performance goal for the CDC processes equal to any online workloads and higher than the Full Load goal. These processes can either be classified using any of the following:
All Versions of Qlik Replicate's R4Z environment
The information in this article is provided as-is and to be used at own discretion. Depending on tool(s) used, customization(s), and/or other factors ongoing support on the solution below may not be provided by Qlik Support.
Installing and upgrading the R4Z product in z/OS
The Microsoft Performance Monitor (perfmon) makes it possible for users to set up automatic data collection jobs which can save data from performance counters at a given interval. This data can, together with Direct Access Agent Logs help to analyze Qlik Direct Access Gateway component and service behavior.
This procedure describes the steps necessary to save such data using Microsoft Performance Monitor Data Collectors.
Steps:
Workload Profiling: To collect performance metrics over a longer period of time, skip steps # 4 and # 7, then delay performing steps # 8 through 10 until you have run a representative daily workload through the gateway. This might take 24-72 hours.
Environment
Qlik Data Gateway - Direct Access all versions.
The information in this article is provided as-is and to be used at own discretion. Depending on tool(s) used, customization(s), and/or other factors ongoing support on the solution below may not be provided by Qlik Support.
Qlik Replicate's Data/Task directory is filling up with an uncompressed temp backup file named UncomprsTemp_Test for Support.bak. A warning is logged in the task log:
[ASSERTION ]W: Failed to write to the decompressed backup file 'UncomprsTemp_Test for Support.bak' (sqlserver_drd_utl_common.c:2363)
Uncheck Replicate has file-level access to the backup log files in the Advanced tab of your connection properties. Note that this option can lead to performance degradation.
With this option checked: Qlik Replicate copies the compressed file onto the Qlik Replicate server and uncompressed the log. Qlik Replicate can then read the uncompressed file directly from the Qlik Replicate server.
With this option unchecked: Qlik Replicate uses the SQL Server (source system) to read the logs so Qlik Replicate reaches out over the network to SQL Server to ask it to read a log and since the logs are on the shared folder SQL Server then needs to reach back to the Qlik Replicate server. This has a performance impact. It will result in latency.
For more information, see Setting advanced connection properties | Qlik Replicate Help.
Increase the disk space for the replicate data directory. We recommend having a minimum of five times the compressed file size readily available. If your compressed file is 100 GB, then you will need a minimum of 500 GB available. In rare scenarios there are instances when uncompressing a file you may need more than 5 times the compressed file size, but this is rare as mentioned before.
Please keep in mind that for Option 2, I am referring to each bak file. Therefore, if you have 8 tasks with a 165 GB file, you'll need a disk with at least 8 (tasks) * 5 (decompression) * 165 GB (bak file size) = ~6,5TB.
In this document we outline common reload configuration and performance best practices for both Qlik Cloud and Qlik Sense Enterprise on Windows.
Users with Professional (Qlik Cloud) or Full User entitlement (Qlik Data Integration) entitlements can:
Administrators (Tenant Admins and Analytics Admins) can:
A default role for Reload Admins in Qlik Cloud is not available. Reload management is currently defined as specified above
For information on how to manage permissions in spaces, see:
Managing permissions in shared spaces
Managing permissions in managed spaces
Qlik Sense Users with the correct Security Rules applied can
Administrators can:
Reloads are intended to refresh data in your analytics apps. Different scenarios may apply when setting up your reloads:
Suggested Best Practice: Determine which users require app and reload permission and apply appropriate entitlements accordingly. See Managing permissions in shared spaces.
If you move an app between Spaces, the reload schedule schedule may be deleted.
Suggested Best Practice: use Application Automation to chain reload tasks. See Task chaining for details.
Suggested Best Practice: Enable reload management from the Qlik Sense hub (August 2022 and later versions only). For detailed information on how to to configure reload management, see Configure Reload Management from the Qlik Sense Hub.
Suggested Best Practice: Use Event Triggers to chain reload tasks. See Qlik Sense App Reloads for details.
The performance advice applies to both Qlik Cloud and Qlik Sense Enterprise on Windows (on premise).
General information: Partial Reload
See How to perform a partial reload in Qlik Sense On Premise.
In both Managed and Shared spaces, Partial Reload can be scheduled in Qlik Cloud using the Reload Scheduler tool. Select the Partial Reload check box:
For limitations around Partial Reloads, see Partial Reload Limitations.
General information: Incremental Reload
When to use Incremental Reload:
The app provides insights on:
The Reload Analyzer will help answer these questions and more!
Read about and Download: Qlik Cloud Reload Analyzer
Reloading app data - Qlik Cloud
Managing reload schedules - Qlik Cloud
Task Chaining - Qlik Cloud
Create Reload Tasks - Qlik Sense Client Managed
Managing Tasks and Triggers - Qlik Sense Client Managed
Creating a Task Chain - Qlik Sense Client Managed
Qlik Sense: Run reload tasks in QMC using API
Troubleshooting - Loading data
Data connections to Qlik Cloud or Qlik Sense Enterprise on Windows can fail for several reasons. In this article we aim to provide you with a step-by-step guide on how to begin troubleshooting data connectivity errors, ruling out the most common root causes.
Content:
The first step in your investigation is to verify basic network connectivity to the database.
Work with your network administrator on both where necessary.
If the network and firewall settings are not the issue, verify if the database operates as expected.
Next, we verify if the credentials and settings in our connection are correct. The easiest way to do this is by logging in directly to the database using its native client.
Examples of on-premise database clients include:
For cloud-based databases:
The specific tools you use will depend on the cloud provider you are using and the type of database you have.
Verify if the following are correct in Qlik Sense (details may vary depending on your data connection):
Depending on your permissions, you can either edit/verify your connection directly in the Qlik Sense App Data Load Editor or in the Space.
Using the Data Load Editor:
Depending on your permissions, you can either edit/verify your connection directly in the Qlik Sense App Data Load Editor or in the Qlik Sense Management Console.
Using the Data Load Editor:
Using the Qlik Sense Management Console:
Once you have confirmed the connection details, you can test the connection in Qlik Sense.
Data connections generally fail during reload. Qlik Sense will log details of the error or preceding warnings in the corresponding reload log. These error and warning messages can then be used to look up solutions or troubleshoot further.
Search the warnings and errors in our knowledge base for known solutions. Begin your search here.
If you have completed all the above and have not found solutions in the Qlik knowledge base, reach out to the database vendor for additional support. They will be able to provide specific information on troubleshooting the issue with your specific database.
The number of connections that Qlik Replicate will use varies depending on the following:
Some of these connections may appear idle for periods of time.
The following is a general description of the number of connections Qlik Replicate will open.
During Full load:
On Source:
Up to 4 X [Tables_loaded_in_parallel] (as defined per table for the selected tables in the task) + 6 connections. This should be a total of 10 connections, for example, if you're replicating one table with no tables being loaded in parallel.
On Target:
Up to 2 X [Tables_loaded_in_parallel] + 6 connections
During CDC (after full load is over):
On Source:
Up to 6 connections
On Target:
Up to 6 connections
The number of connections might increase when replicating LOB columns due to connections needed to perform source lookups due to incomplete LOB data written to source database log files, and to apply this data to the target.