Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.
Search our knowledge base, curated by global Support, for answers ranging from account questions to troubleshooting error messages.
You have been running Qlik Sense normally for quite some time. Over time you have accumulated an abundance of reload tasks configured in your Qlik Sense QMC.
Recently however, you have noticed that Qlik Sense QMC reload tasks are in the following state:
To resolve the issue, it is recommended to add an additional scheduler node (or nodes) in order to manage the ever increasing number of reload tasks in the affected Qlik Sense environment.
As time goes on, add additional scheduler nodes in proportion to the increasing number of reload tasks added/deployed in your environment.
The existing Qlik Sense scheduler nodes simply cannot manage the additional burden placed upon it by ever increasing reload tasks.
Concurrent Reload Settings in Qlik Sense Enterprise
QB-20013
Qlik Replicate 2023.5.0.284 has been observed to contain a behavior where source and target latency are miscalculated leading to big differences of fluctuating latency estimates. This behavior is resolved in a new patch.
2024-06-11T09:04:24 [PERFORMANCE ]T: Source latency 184.42 seconds, Target latency 12256.62 seconds, Handling latency 12072.21 seconds (replicationtask.c:3789)
2024-06-11T09:04:54 [PERFORMANCE ]T: Source latency 214.42 seconds, Target latency 214.42 seconds, Handling latency 0.00 seconds (replicationtask.c:3789)
2024-06-11T09:09:24 [PERFORMANCE ]T: Source latency 484.43 seconds, Target latency 12480.20 seconds, Handling latency 11995.77 seconds (replicationtask.c:3789)
This bug has been resolved in Qlik Replicate 2023.11 Service Pack 04 and Qlik Replicate 2024.05 General Availability release
QB-27442
RECOB-7562
This article provides a list of the best practices for Qlik Sense configuration. It is worth implementing each item, especially for a large environment so that your database can handle the volume of requests coming from all its connected nodes.
For basic information, see Max Connections.
Specifies the maximum number of concurrent connections (max_connections) to the database. The default value for a single server is 100.
In a multi-node environment, this should be adjusted to the sum of all repository connection pools + 20. By default, this value is 110 per node.
Assuming two nodes and assuming the default value of 110 per node, the value would be 240.
The value of 110 above is a default example. You can further refine the value.
The connection pool for the Qlik Sense Repository is always based on core count on the machine. To date, our advise is to take the core count of your machine and multiply it by five. This will be your max connection pool for the Repository Service for that node.
This should be a factor of CPU cores multiplied by five.
If 90 is higher than that result, leave 90 in place. Never decrease it.
For more information about Database Max Pool Size Connection, see https://wiki.postgresql.org/wiki/Number_Of_Database_Connections
Optimizing Performance for Qlik Sense Enterprise
PostgreSQL: postgresql.conf and pg_hba.conf explained
Database connection max pool reached in Qlik Sense Enterprise on Windows
Reloads with Data Gateway are randomly slower, sometimes they last for 3 hours until they are automatically aborted.
The same reloads usually run faster and they can be completed in the right time if relaunched manually or automatically.
The logs do not show specific error messages.
We recommend two actions to resolve the problem.
The first is to activate the process isolation in order to reduce the number of reloads running at the same time. Please, follow this article.
It is possible to start with a value of 10 for the ODBC|SAPBW|SAPSQL_MAX_PROCESS_COUNT parameter and adjust it after some tests.
The second action is to add the "DISCONNECT" command after every query and to start every new query with a "LIB CONNECT".
This will force the closure and re-creation of the connection every time it is needed.
More information about the DISCONNECT statement can be found here.
We always recommend to keep Data Gateway on the latest version available.
This intermittent problem can be due to different causes.
In many cases the system can't handle multiple connections efficiently and this can lead to severe slowness in the data connection. Activating a Process Isolation will help to avoid this.
It is also possible that there is a delay between the connection opening and the query.
A connection for a query can be opened at the beginning a reload, then kept open for a while and re-called later for another table load in the script.
It is possible that there is a disconnection when the connection is not working. This can happen if another connection to the same location is called by a concurrent reload or if a timeout automatically closes the connection.
It is possible to force Data Gateway to recreate the connection using the "DISCONNECT" statement in the script.
Key Split is a performance tuning option within Qlik Gold Client. Introduced in Qlik Gold Client 8.7.3, it is used to control which tables allow for splitting of table content in different Data Flows for export/import. This functionality is triggered when the configured table is a Header table in a Data Type, where it will split the table entries.
The benefit is to allow Gold Client to parallelize the import of data in the same table using multiple jobs, making the import process much faster.
Key Split is to be used carefully and for very specific export/import scenarios, especially when there is a performance issue importing tables on a target system. To maximize the functionality, parallel processing option must be used.
At this moment, some special table type fields like GUID, RAW, STRING, XSTRING, etc. should not be used with Key Split.
There are 4 different types of Key Split options available within Qlik Gold Client:
For a specific range of documents, it will separate the documents by the first alphanumeric digit and create a Container for each of them.
A, B, C, …, 0, 1, 2, …, Special Chars.
Example: B123456789 (this entry will be added with all that start with “B”)
For a specific range of documents, it will separate the documents by the first alphanumeric digit grouping and creating a Container for each of them.
A-C, D-F, G-I, J-L, M-O, P-R, S-U, V-X, Y-Z, 0-2, 3-5, 6-8, 9-Special Chars.
Example: B123456789 (this entry will be added with all that start with “A* B* or C*”)
For a specific range of documents, it will separate the documents by the last numeric digit and create a Container for each of them.
0, 1, 2, 3, 4, 5, 6, 7, 8, 9
Example: 123456789 (this entry will be added with all which last digit is “*9”)
For a specific range of documents, it will separate the documents by the last two numeric digits grouping and creating a Container for each of them.
Group 1: 00-04
Group 2: 05-09
Group 3: 10-14
…
Group 20: 95-99
Example: 1234567899 (this entry will be added with all which last digits are between “*95- 99”)
This diagram shows the Key Split logic for a set of Materials using the Alphanumeric and Alphanumeric by Grouping options (based on the first alphanumeric value):
This diagram shows the Key Split logic for a set of Materials using the Numeric and Numeric by Grouping options (based on the last numeric digit):
Tables using this logic are configured in the Configuration > Data Framework > Additional Tools > Key Split Config
To activate the key split in Gold Client, it is necessary to provide the Table name, Field name and Key split type. The Active/Inactive flag allows to enable/disable the functionality.
It is possible to deactivate Export Engine functionality to split table exports into multiple files based on keys in the Configuration > Administration > MConfig
Splitting the export of MARA entries based on material Number – MATNR.
In the case of exporting MM – MATERIAL MASTER using Parallel Processing, for a specific range of documents, it will separate the documents by the first digit and create Containers for each of them.
During the import process, this logic will allow the import of data in parallel to MARA table faster.
Qlik Gold Client Sizing Report for table MARA Export with Key Split inactive
Qlik Gold Client Sizing Report for table MARA Export with Key Split Alphanumeric active
Qlik Gold Client Sizing Report for table MARA Export with Key Split Alphanumeric by Grouping active
Splitting the export of ACDOCA entries based on document Number - BELNR.
In the case of exporting FI - FINANCE DOCUMENTS using Parallel Processing, for a specific range of documents, it will separate the documents by the last digit(s) and create Containers for each of them.
During the import process, this logic will allow the import of data in parallel to ACDOCA table faster.
Qlik Gold Client Sizing Report for table ACDOCA Export with Key Split inactive
Qlik Gold Client Sizing Report for table ACDOCA Export with Key Split Numeric active
Qlik Gold Client Sizing Report for table ACDOCA Export with Key Split Numeric by Grouping active
Please refer to the Qlik Gold Client Configuration and Utilities - User Guide for more information.
Thanks to Qlik SAP Solutions Engineer Hugo Martins for drafting this content!
NPrinting report is hanging on preview or in a publish task preventing the report from generating as expected. In many cases with 'Carded' Qlik Sense themes, we are seeing this problem.
This is also seen when the extended sheet feature is used
"error=task SENSE_JS_PAINT_COVER_POINTS_TIMEOUT"
There is currently a fix coming to resolve 'carded' custom themes however this fix is still under investigation.
Information provided on this defect is given as is at the time of documenting. For up to date information, please review the most recent Release Notes, or contact support with the QB-24997 for reference.
Possible workarounds for this are:
NOTE: Extended or Custom sheet sizes are not fully supported. See Qlik Help page below for details
Qlik Sense custom and extended sheets
After upgrading Qlik Enterprise Manager to 2022.11, increased connection errors to the Qlik Replicate server can be observed, with the connection terminating approximately every 30 minutes.
The following error is logged:
3 2023-12-19 06:09:06 [ServerDto ] [ERROR] Failed to get monitoring data for server: 172.29.73.29. Message:'SYS-E-SESS_LOST, The client is not connected to the Server. Please connect and then try again.'''.
This is a known defect. The fix will be included in the next possible Qlik Enterprise Manager release (tentatively 2023.11 SP03).
Fix Description: Added auto-reconnect flag where it was missing to allow QEM to better manage the connection drops and attempt to re-establish the EM<-->QR communication, especially in high-load situations.
The following are test patches that address this issue. Please contact support to obtain the test patch.
Qlik Enterprise Manager 2022.11.0.634
Qlik Sense Enterprise on Windows
Result files cannot be found. Depending on which tool is used (Exerciser, Gopherciser or JMeter), the extension and format is different. At the time of writing this article (March 2024), the latest version of Qlik Sense Enterprise scalability tools is using Gopherciser by default.
Set the extension for the file in the "Main" tab (either .log or .tsv)
In the folder structure linked to the data connection "DataFiles", make sure that the result file has the correct extension (.tsv in this example) and is placed in the Gopherciser folder.
The TimeSlot field is created based on the time field (case-sensitive), make sure this column is included in the result file.
//Tab: __SUB LoadGopherciser
Load
'Gopherciser' as Origin,
FileDir() as path,
Timestamp(Timestamp#(Timestamp(Mid(Replace("time",'T', ' '),1,16),'YYYY-MM-DD hh:mm') & ':00', 'YYYY-MM-DD hh:mm:ss')) as TimeSlot, // Local time
"time" as Org_TimeStamp,
...
Scalability Results Analyzer
Qlik Sense Enterprise Scalability Tools
A Qlik Replicate task can generally be stopped from the Qlik Replicate web console. However, sometimes stopping the task takes longer than anticipated. A delay in the task being stopped can be caused by, for example:
If you plan to reload the task or to start it from a timestamp, you have the option to kill or terminate the task at the operating system level, as the task will not need to reference its stop position to resume where it left off. A reload or start from a timestamp is considered a fresh start of the task.
Never manually terminate or kill the task if you intend to resume the task and you do not see this entry in the task log:
00009760: 2022-05-23T23:30:49 [UTILITIES ]I: The last state is saved to file 'D:\Program Files\Attunity\Replicate\data\tasks\TASK_NAME/StateManager/ars_saved_state_000002.sts' at Tue, 24 May 2022 04:30:41 GMT (1653366641301364) (statemanager.c:642)
After the task has been terminated, please check with your source and target DBA to ensure any orphaned connections are ended.
Qlik Replicate, all versions
Windows
The information in this article is provided as-is and to be used at own discretion. Depending on tool(s) used, customization(s), and/or other factors ongoing support on the solution below may not be provided by Qlik Support.
After upgrading to Qlik Sense Enterprise on Windows May 2023 (or later), the Qlik Sense Repository Service may cause CPU usage spikes on the central node. In addition, the central Engine node may show an increased average RAM consumption while a high volume of reloads is being executed.
The Qlik Sense System_Repository log file will read:
"API call to Engine service was successful: The retrieved 'StaticByteSize' value for app 'App-GUID' is: 'Size in bytes' bytes."
Default location of log: C:\ProgramData\Qlik\Sense\Log\Repository\Trace\[Server_Name]_System_Repository.txt
This activity is associated with the ability to see the Base memory size in the Qlik Sense Enterprise Management Console. See Show base memory size of apps in QMC.
The feature to see Base memory size can be disabled. This may require an upgrade and will require downtime as configuration files need to be changed.
Take a backup of your environment before proceeding.
If any issues occur, revert the changes by restoring the backed up files and open a ticket with Support providing the changed versions of repository.exe.config, capabilities.json mentioning this article.
Show base memory size of apps in QMC
QB-22795
QB-24301
Qlik Sense Enterprise on Windows May 2023, August 2023, November 2023, February 2024.
It is finally here: The first public iteration of the Log Analysis app. Built with love by Customer First and Support.
"With great power comes great responsibility."
Before you get started, a few notes from the author(s):
Chapters:
01:23 - Log Collector
02:28 - Qlik Sense Services
04:17 - How to load data into the app
05:42 - Troubleshooting poor response times
08:03 - Repository Service Log Level
08:35 - Transactions sheet
12:44 - Troubleshooting Engine crashes
14:00 - Engine Log Level
14:47 - QIX Performance sheets
17:50 - General Log Investigation
20:28 - Where to download the app
20:58 - Q&A: Can you see a log message timeline?
21:38 - Q&A: Is this app supported?
21:51 - Q&A: What apps are there for Cloud?
22:25 - Q&A: Are logs collected from all nodes?
22:45 - Q&A: Where is the latest version?
23:12 - Q&A: Are there NPrinting templates?
23:40 - Q&A: Where to download Qlik Sense Desktop?
24:20 - Q&A: Are log from Archived folder collected?
25:53 - Q&A: User app activity logging?
26:07 - Q&A: How to lower log file size?
26:42 - Q&A: How does the QRS communicate?
28:14 - Q&A: Can this identify a problem chart?
28:52 - Q&A: Will this app be in-product?
29:28 - Q&A: Do you have to use Desktop?
Qlik Sense Enterprise on Windows (all modern versions post-Nov 2019)
*It is best used in an isolated environment or via Qlik Sense Desktop. It can be very RAM and CPU intensive.
The information in this article is provided as-is and to be used at own discretion. Depending on tool(s) used, customization(s), and/or other factors ongoing support on the solution below may not be provided by Qlik Support.
Optimizing Performance for Qlik Sense Enterprise - Qlik Community - 1858594
QlikView and Qlik Sense files are like ZIP file, as in they are highly compressed. So obtaining an estimate of how much RAM is needed for loading Apps into memory (footprint) is difficult to provide across the board. The expected memory usage associated with the app can also depend on how many users access the app and how much they are used.
An estimate of the RAM needed per app can be built on the below, but for accuracy always test by loading the app into memory and using the Qlik Scalability Tools to obtain a baseline of memory usage for each app as it is accessed by the foreseeable number of users.
Telemetry logging which allows for using the Telemetry Dashboard, is also an available suitable tool.
Since the release of the February 2019 version, Sense System Performance Analyzer monitoring App can be used to determine app footprint as well. Since the release of the June 2018 version, App Metadata Analyzer monitoring App can also be used. However, these last three may not provide the same data as the Scalability Tools.
An alternative is to use a test environment and simply observe RAM usage increases as apps are first opened (footprint), and as number of users accessing the app increase, then as number of operations are performed within the app. The test environment results for observed memory and CPU usage can be recorded as a baseline for the particular app in production, which assists in determining future sizing/scaling needs.
Need direct assistance in evaluating your Qlik Sense and QlikView apps? Qlik's Professional Services are available to assist you.
RAMInitial = SizeOnDisk × FileSizeMultiplier ; this is the initial RAM footprint for any application
FileSizeMultiplier: range between 2-10 (this is a compression* ratio depending on the data contained in the app)
*Compression is based upon the data, and how much we can compress depends upon the homogeneity of the data. The more homogeneous, the more compression Qlik can achieve.
More information on one way of optimizing a document can be found on the Qlik Design Blog: Symbol Tables and Bit-Stuffed Pointers
RAMperUser = RAMinitial × userRAMratio ; this is the RAM per each incremental user
userRAMratio: range between 1% -10%
Total RAM used per app :
TotalRAM = (RAMperUser × Number of users) + RAMinitial
Example:
SizeOnDisk = 1 GB
File Size Multiplier 6 (range is from 2 to 10) This is an example value. A value can be obtained by reviewing telemetry (how far does the app expand?) or opening the app in a QlikView or Qlik Sense desktop and measuring the expansion based on how large it grows in memory.
RAMinitial = 1 * 6 = 6 GB
If we take a RAM Ratio per User of 6% then it is 0.06 (range is from 1 to 10)
RAMperUser = 6 x 0.06 = 0.36GB
Then RAM Required for 30 Users
TotalRAM = (RAMperUser × No. users) + RAMinitial
TotalRAM = (0.36 * 30) + 6
TotalRAM = 16.8 GB
NUMA, Non-Uniform Memory Access, is a computer memory design used in Multiprocessing, where the memory access time depends on the memory location relative to a processor. Under NUMA, a processor can access its own local memory faster than non-local memory, that is, memory local to another processor or memory shared between processors.
QlikView generally performs better with NUMA disabled. The NUMA setting is controlled by the computer BIOS, and the recommendation is to disable it for best possible performance.
Some hardware does not allow disabling NUMA. For these scenarios QlikView 11.20 SR2 and later can automatically detect the presence of NUMA and if present adjust the QVS.exe to adapt to a NUMA environment. While not as efficient as disabling NUMA in BIOS, it restores some of the performance lost when NUMA is enabled.
The automatic detection only works in 2 or 4-socket systems. For 8 socket machines, the NUMA detection fails, leaving the QVS.exe to believe itself to exist in a non-NUMA environment.
The software NUMA adaption is controlled with the EnableNumaRoundRobin variable in the QlikView Server settings.ini file.
All recommendations can be found on the community: Quick tips #8 - Server settings for best performance they are being updated if Performance & Scalability team sees that changes in server configurations impact the performance. So it worth checking them from time to time.
Please keep in mind that the Performance & Scalability team tries to test many hardware configurations, but it is still a very small subset of possible server configurations seen in customer deployments. Due to this, in case of doubt, they suggest testing the impact of the setting. For example, one customer might on average see that for their deployment disabling NUMA in BIOS is beneficial, while others might not notice any difference.
All changes should be modified with caution, because set NUMA (in hardware and software) improperly might result in (significant) performance degrade, or fatal memory issue in some rare case.
EnableNumaRoundRobin is a setting configurable in the settings.ini file.
0 (default): Automated state where QlikView Server at startup detects whether NUMA is enabled or disabled at certain hardware and adapts to it. If NUMA is enabled QlikView Server will ignore NUMA nodes when allocating memory.
1: QlikView Server is acting as if NUMA is enabled from BIOS (without any attempts to detect it)
2: QlikView Server will not ignore NUMA nodes if present (i.e. NUMA enabled)
Modify the settings.ini and add EnableNumaRoundRobin=X, where X is your desired value. The settings.ini file is (by default) stored in %ProgramData%\QlikTech\ on your QlikView server node.
For information on how to change the settings.ini, see How to Modify the QlikView Server Settings.ini file.
A similar method can be used in the QlikView Desktop client. The settings used are slightly different.
EnableNumaRoundRobin (integer type parameter) for QlikView Desktop:
0 (default value): detect NUMA automatically and enable it in QlikView if possible
1: force QlikView Desktop to enable NUMA
Other: force QlikView Desktop to disable NUMA
For details of instruction and different testing result please refer to attached pdf file.
Qlik Replicate SAP Extractor Endpoint can be tuned to improve the performance of data being processed from an SAP environment. This is achieved by adding an internal parameter: sqlConnector. To confirm the performance improvements of the task, first record the task duration before implementing the parameter, then compare it to the subsequent runs.
Before the internal parameter can be added to the Qlik Replicate Extractor Endpoint, the below SAP Transports must be installed:
Once the transports have been installed make sure you follow the below steps to set the added Permissions on the SAP Replicate User:
Add the following authorization objects and settings to the RFC user ID:
Do the following:
For more information on how to import/upgrade the sqlConnector Transport, see Importing the SAP transports.
Configure the SAP Extractor Endpoint by defining the useSqlConnector Internal Parameter.
SAP Transports
Importing the SAP transports
After an upgrade to Qlik Sense Enterprise on Windows February 2023 or later, reloads may take longer to complete.
Example:
A Qlik Sense test script app (CTRL+00) may have an increased reload time of two minutes, rather than the expected two to ten seconds.
The issue was identified and mitigated in collaboration with Merck Group and Cognizant (Thiago Cunha, Akash Pohare, Vincent Wei), alongside Qlik's internal development and support teams (Vijay Vekariya, Oystein Kolsrud, Sebastian Linser, Eugene Sleator).
Disable SaveScriptWithSetScript. This disabled any additional save script calls and speeds up the reload.
To do so:
[Settings 7]
SaveScriptWithSetScript=0
In February 2023, Qlik introduced a collaborative feature allowing co-development in load scripts. This feature is intended for Qlik Cloud only. See Collaborate on load scripts in shared spaces. It can be safely disabled in an on-premise installation.
As of the February 2024 release, the feature will be disabled by default.
QB-23619
QB-22810
Qlik Sense Enterprise on Windows February 2023 - November 2023
Qlik Catalog Data loads fail to complete due to an out of memory error.
The issue can be addressed by either increasing memory on the host running Qlik Catalog and engine or by decreasing the number of concurrent loads Catalog will process (32 by default). You may need to do both.
Decrease the number of concurrent loads to 4.
The setting is changed in the core_env.properties file located in the directory: QDC_HOME/<Tomcat>/conf/core_env.properties.
A Tomcat restart is required after making the change in the core_env.properties file.
Make the change when no loads are running. Find and edit the following (do not add a new entry):
# Number of Java threads in pool used for each of loading data, prepare and publish.
# Restart required. Default: 32
hadoop.job.poolsize=4
If the loads are successful, increase the count gradually and continue testing.
If it is easy to provision more RAM, consider doing so.
Qlik Catalog uses the "engine" container to convert QVDs to CSVs (in order to sample and profile the data). The issue is that engine is running out of memory attempting to concurrently process several QVDs. Sometimes engine gracefully handles the memory exhaustion and returns an error to Qlik Catalog. Other times engine crashes and the container is restarted. In both cases, the job status is FAILED in Qlik Catalog.
If you experience delayed or stuck automations in the Blendr.io platform, please note that we are evaluating measures to address these issues, such as code updates and increased capacity. However, due to the previously announced end of life of the product, a long-term solution may not be possible.
When you encounter a stalled automation, you can still open a support case. Support will then manually set the automation to failed in order to allow any queued automations to continue.
Further, for troubleshooting purposes, please note the following:
If Oracle is configured with a Physical Standby, and the Oracle LOG_ARCHIVE_DEST_n initialization parameter is configured with the DELAY option, running a Full Load and Apply Changes task on an Oracle Physical Standby instance may result in missing or erroneous data in the target database.
The following warning may be thrown in the log files:
Using standby database could have caused the missing events problem during Full Load and CDC synchronization.
If you do not have the DELAY parameter configured and there is no data loss, this warning message can be ignored. Please check with your DBA if the DELAY parameter is set.
This issue can be resolved using an internal parameter: standbyDelayTime, which should be set to a value little bit bigger than the Delay parameter of LOG_ARCHIVE_DEST_n initialization parameter. Units are in minutes.
For information on how to set internal parameters, see Qlik Replicate: How to set Internal Parameters and what are they for?
This solution may lead to redundant records for a table without PK/UI if there were INSERT operations during that period. Example - LOG_ARCHIVE_DEST_2 = 'DELAY=10'
When installing and using the DB2 z/OS endpoint in Qlik Replicate, WLM tasks on the mainframe may get cancelled. There are multiple root causes for this, but a common one can be addressed by reviewing the execution priority (WLM settings).
The Full Load processes will utilize any available ZIIP processors. The best practice is to set the performance goal so that the processes get classified as lower than online workloads. In a very busy system, you might consider setting Honor Priority to NO for the service class assigned to the Full Load. This requires z/OS 2.1 + OA50845 or z/OS 2.2 and above.
Best practice is to set the performance goal for the CDC processes equal to any online workloads and higher than the Full Load goal. These processes can either be classified using any of the following:
All Versions of Qlik Replicate's R4Z environment
The information in this article is provided as-is and to be used at own discretion. Depending on tool(s) used, customization(s), and/or other factors ongoing support on the solution below may not be provided by Qlik Support.
Installing and upgrading the R4Z product in z/OS