
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Qlik Catalog Release Notes - May 2022 Initial Release to Service Release 1
Table of Contents
- What's new in Qlik Catalog May 2022
- Publish to Qlik Sense: Multiple Sense Server Support in User Interface
- Lineage API Additions to Enable Updating "other" Nodes and Edges
- New Sources Auto-Configure to Allow Non-Ascii and Control Characters
- New Installs Automatically Enable Tomcat HTTPS and NextGen XML
- Post-Install Script to Auto-Configure Container Services
- Resolved Defects
- May 2022 SR1 (4.14.1)
- Improved Qlik Sense Integration Scripts
- Fixed Incorrect Prepare Dataflow Script Generation
- Fixed "Host" Header Poisoning
- Fixed Install to Alternate Home Directory
- May 2022 Initial Release (4.14)
- Fixed Double-escaping of Backslash When Loading Data
- Addressed Spring Framework Vulnerabilities
- No Longer Supported
- Upgrade notes
- Migrating to or Upgrading Tomcat 9
- Process if Upgrading From June 2020 or Earlier
- Log of Changes to File core_env.properties
- May 2022 SR1
- May 2022
- February 2022 SR1
- February 2022
- November 2021 SR2
- November 2021 SR1
- Downloads
The following release notes cover the versions of Qlik Catalog released in May 2022.
What's new in Qlik Catalog May 2022
- Publish to Qlik Sense: Multiple Sense server support in user interface
- Lineage API additions to enable updating "other" nodes and edges (e.g., rename)
- New Sources auto-configure to allow non-ascii and control characters
- New installs automatically enable Tomcat HTTPS and NextGen XML
- Post-install script to auto-configure container services
Publish to Qlik Sense: Multiple Sense Server Support in User Interface
Multiple Sense servers may now be configured for Publish to Qlik Sense. Specific Sense servers are configured using the Catalog user interface, along with traditional QVD Import settings (see menu Admin / Qlik Sense). Only global Publish to Qlik Sense settings remain in file core_env.properties. When Publish to Qlik Sense is selected for QVD entities, the Sense server from which the QVDs originated is automatically selected. When non-QVD entities are selected, a new dialog is shown and the user must pick the Sense server on which the app will be created/updated.
Please see the online help, as well as the "Qlik Catalog + Qlik Sense Integration Guide", for more detail. In addition, impacted core_env properties are listed in the appendix.
Dialog to select Qlik Sense server when publishing non-QVD entities:
Dialog to configure Publish to Qlik Sense (along with QVD Import):
Lineage API Additions to Enable Updating "other" Nodes and Edges
Two additional update (PUT) calls were added to the lineage API:
- update node label (aka name) and metadata
- update edge metadata
Only nodes and edges of type "other" may be updated. These are typically used to illustrate upstream lineage ancestry (and are created/deleted using the API). Until this enhancement, nodes could not be renamed.
Please visit the "live" documentation included with Catalog at Support / API Documentation.
New Sources Auto-Configure to Allow Non-Ascii and Control Characters
Going forward, newly created Sources (of all types except MAINFRAME) will automatically be configured to allow non-ascii and control characters. A common use case that will benefit is loading JDBC (relational) data containing foreign characters. These loads will now become "zero-config".
Because these properties are set at the Source level, they may be overridden at the Entity level. To revert to legacy behavior, either delete the properties or set them to false (see screenshot). Note that the Discover module version of the Source also has these properties set. Prepare Dataflow target Entities added to these Sources will also pick up the settings.
New Installs Automatically Enable Tomcat HTTPS and NextGen XML
Going forward, new Catalog installs are automatically configured to use HTTPS (port 8443). HTTP (port 8080) is automatically redirected to HTTPS. The NextGen XML feature (used to load XML data) is automatically configured to work alongside (a dedicated HTTP port, 8082, is configured for it).
Post-Install Script to Auto-Configure Container Services
A new script, QDCpostinstall.sh, may be run after installing Catalog -- it configures all containers as auto-start Linux services.
There are now three scripts that can be used for the entire installation process:
- QDCprereqs.sh -- run as root, installs all prerequisite software, creates qdc account, etc.
- QDCinstaller.sh -- run as service user (qdc), installs Catalog
- QDCpostinstall.sh -- run as root, configures Linux services
Please see the installation guide for more details.
Resolved Defects
- QDCB-1148 - Fixed Incorrect Prepare Dataflow Script Generation
- QDCB-1149 and QDCB-1150 - Improved Qlik Sense Integration Scripts
- QDCB-404 - Fixed "Host" Header Poisoning
- QDCB-1154 - Fixed Install to Alternate Home Directory
- QDCB-1143 - Fixed Double-escaping of Backslash When Loading Data
-
QDCB-1139 - Addressed Spring Framework Vulnerabilities
May 2022 SR1 (4.14.1)
Improved Qlik Sense Integration Scripts
Jira ID: QDCB-1149 and QDCB-1150
The Node.js scripts used to integrate Qlik Catalog with Qlik Sense where enhanced to uniformly:
- perform request authorization -- not all scripts validated that the Qlik Sense SSL certificate was signed by the root.pem CA certificate copied to the Catalog server
- configure a network timeout -- many scripts did not implement a timeout handler, leading to timeouts of up to two minutes
- handle hostname mismatch -- for some scripts, return value ERR_TLS_CERT_ALTNAME_INVALID was seen if the host which was connected to was not in the SSL certificate's altnames
To disable enhanced authorization, edit /usr/local/qdc/qlikpublish/securityUtils.js and change the following to false:
exports.rejectUnauthorized = true;
exports.rejectUnauthorizedForCreateAndUpdateApp = true;
Fixed Incorrect Prepare Dataflow Script Generation
Jira ID: QDCB-1148
Prepare Dataflow execution failed due to incorrect script generation for certain entity fields used in Aggregation functions. If an entity contained two fields (e.g., QTY_BASE_PRICE_SIGN and BASE_PRICE), where one field was a substring of the other, script generation was incorrect if the longer of the two fields was used in an Aggregation function (a double replacement occurred).
Fixed "Host" Header Poisoning
Jira ID: QDCB-404
An attacker could craft a request to Catalog containing an HTTP Host header value of a site they control causing Catalog to return a page with references to their malicious site. HTTP caching mechanisms between Catalog and its clients can become poisoned with malicious content serving the attacker's page in place of Catalog's intended content.
The Host header value should not automatically be trusted by Catalog. Instead, a protected configuration setting should be used to specify the desired value. A new core_env property is introduced:
# To protect against HTTP headers such as "Host" and "X-Forwarded-Host" being abused by a malicious client, set this
# to the hostname used by clients. It is embedded in link results returned by Catalog searches (e.g., /catalog/search).
# Restart required. Default: not set.
external.hostname=<FULLY_QUALIFIED_DOMAIN_NAME>
For new installs, this property is initialized to the result of Linux command "hostname". Upgrades must manually add this property. Also, upgrades must manually update webapps/qdc/WEB-INF/web.xml and add the following before the closing "</web-app>" tag:
<filter>
<filter-name>ServerNameRequestFilter</filter-name>
<filter-class>com.nvs.ui.security.ServerNameRequestFilter</filter-class>
</filter>
<filter-mapping>
<filter-name>ServerNameRequestFilter</filter-name>
<url-pattern>/*</url-pattern>
<dispatcher>REQUEST</dispatcher>
<dispatcher>FORWARD</dispatcher>
</filter-mapping>
To specify legacy behavior, either remove the filter definition from webapps/qdc/WEB-INF/web.xml or do NOT set the core_env property 'external.hostname'.
Fixed Install to Alternate Home Directory
Jira ID: QDCB-1154
Catalog was not correctly configured when an alternate home directory (e.g., /apps/qdc rather than /usr/local/qdc) was specified. This has been fixed. To install to an alternate home directory, change QDC_HOME and run the following:
- QDCprereqs.sh -- change QDC_HOME and run as root/sudo user
- QDCinstaller.properties -- change QDC_HOME and then run QDCinstaller.sh as service account (qdc)
- QDCpostinstall.sh -- change QDC_HOME and run as root/sudo user
Please see the install guide for more information about editing and executing these files.
May 2022 Initial Release (4.14)
Fixed Double-escaping of Backslash When Loading Data
Jira ID: QDCB-1143
If customer data contained escaped characters (e.g., \t), when loading data (under specific configurations) Catalog would add a second backslash (e.g., \\t). If then run through a Prepare Dataflow, the backslashes were again escaped (e.g., \\\\t). This issue has been fixed.
Addressed Spring Framework Vulnerabilities
Jira ID: QDCB-1139
This release addresses the following Spring Framework vulnerabilities:
- CVE-2022-22965 - Spring Framework RCE via Data Binding on JDK 9+
- CVE-2022-22950 - Spring Expression DoS Vulnerability
No Longer Supported
- With the August 2021 release, CDH and HDP Hadoop cluster environments are no longer supported (supported Hadoop cluster environments are AWS EMR and CDP Private Cloud 7)
- With the May 2021 release, Microsoft Internet Explorer 11 is no longer supported and Workflow Scheduler was removed
Upgrade notes
Migrating to or Upgrading Tomcat 9
Beginning with the May 2021 release, only Apache Tomcat 9 is supported. The installer will prohibit other versions. If using Tomcat 7, please first initiate a migration to Tomcat 9 before installing this release. Then, when installing, the upgrade option (-u) is NOT used.
These instructions may also be used to upgrade from an older version of Tomcat 9 to a newer version.
Step |
Sample Commands |
Shutdown and rename old Tomcat 7 or 9 |
cd /usr/local/qdc (or cd /usr/local/podium) ./apache-tomcat-<OLD_VERSION>/bin/shutdown.sh mv apache-tomcat-<OLD_VERSION> old-apache-tomcat |
Download and expand Tomcat 9 - NOTE: adjust version 9.0.62 to use latest 9.0.x series |
wget https://archive.apache.org/dist/tomcat/tomcat-9/v9.0.62/bin/apache-tomcat-9.0.62.tar.gz tar -xf apache-tomcat-9.0.62.tar.gz rm apache-tomcat-9.0.62.tar.gz |
Copy core_env.properties from old Tomcat to new Tomcat 9 |
cp old-apache-tomcat/conf/core_env.properties apache-tomcat-9.0.62/conf/ |
If migrating from Tomcat 7: Extract server.xml from podium.zip and copy to new Tomcat |
unzip -j podium-4.<VERSION>-<BUILD>.zip podium/config/tomcat9-server.xml -d . mv ./tomcat9-server.xml apache-tomcat-9.0.62/conf/server.xml |
If upgrading Tomcat 9: Copy server.xml from old Tomcat 9 to new Tomcat 9 |
cp old-apache-tomcat/conf/server.xml apache-tomcat-9.0.62/conf/ If the old Tomcat 9 was configured for HTTPS, and the keystore (jks file) was stored in the old Tomcat directory, migrate it to the new Tomcat directory, and update conf/server.xml to reference it. Consider placing the keystore file in a non-Tomcat directory such as /usr/local/qdc/keystore. |
Configure QDCinstaller.properties for Tomcat 9 |
Whether using an existing QDCinstaller.properties file from a previous install, or configuring one for the first time, ensure that it is updated to point to Tomcat 9: TOMCAT_HOME=/usr/local/podium/apache-tomcat-9.0.62 |
Finally, when the installer is run, do NOT specify upgrade mode (-u), as some files should be created as if it were a first-time install. |
./QDCinstaller.sh |
At this point, Tomcat 9, if newly installed, will support only HTTP on port 8080.
Verify successful Qlik Catalog startup and basic functionality.
Additional configuration will be required to enable HTTPS on port 8443, apply security headers, etc. If Tomcat 7 used HTTPS, the keystore (jks file) containing the public-private keypair should be copied to Tomcat 9 and conf/server.xml updated.
In addition, Tomcat 7 may have been configured as a service. It should be disabled. Tomcat 9 may be configured as a service to automatically start.
Please see the install guide for guidance on both.
Process if Upgrading From June 2020 or Earlier
Do not attempt to upgrade until the following is understood
If upgrading from a version of Qlik Catalog prior to September 2020 (4.7) there are utilities that MUST be run after Catalog is upgraded. Once run, the utilities need never be run again.
The server may not start until the first two utilities have been run and will log a WARN at startup until the third is run. Do NOT upgrade the server until familiar with these utilities and the information required to run them. It will take time to gather this information. Gathering the information BEFORE Catalog is upgraded will minimize downtime.
Run the utilities in this order:
- jwt2CertsUtility -- please review readme.txt
This will be required if Qlik Sense Connectors have been defined to load QVDs.
Will need to gather networking info and certificate files from Qlik Sense servers.
May be run from any directory.
- singleNodeUpgradeForEntitiesWithBadOrUglyData.sh -- please review comment in script
This will be required only if the installation is single node.
Will need podium_dist database info if defaults altered.
May be run from any directory.
- singleNodeUpgradeToGrantReadOnlyUserAccessToDistSchemas.sh -- please review comment in script
This will be required only if the installation is single node.
Will need podium_dist database info if defaults altered.
May be run from any directory.
Log of Changes to File core_env.properties
A chronological listing (most recent first) of additions, changes in behavior, and deletions to the primary global configuration file, core_env.properties.
May 2022 SR1
ADDITION: Fixed "Host" Header Poisoning
The Host header value should not automatically be trusted by Catalog. Instead, a protected configuration setting is introduced. See earlier description of QDCB-404.
# To protect against HTTP headers such as "Host" and "X-Forwarded-Host" being abused by a malicious client, set this
# to the hostname used by clients. It is embedded in link results returned by Catalog searches (e.g., /catalog/search).
# Restart required. Default: not set.
external.hostname=<FULLY_QUALIFIED_DOMAIN_NAME>
May 2022
CHANGE: Enable Publish to Qlik Sense by Default
Publish to Qlik Sense will now be enabled by default.
# Setting this to true will start showing 'Publish to Qlik' option in Podium UI cart checkouts.
# Default: true
is.publish.to.qlik.enabled=true
DELETION: Publish to Qlik Sense: Multiple Sense Server Support in User Interface
Configuration of specific Sense servers for Publish to Qlik Sense has been moved to the Catalog user interface. General settings remain in core_env.properties.
The following is taken from the "Qlik Catalog + Qlik Sense Integration Guide". The nine, new UI fields for Publish to Qlik Sense were formerly in core_env.properties. These properties can now be retired (either commented-off or removed from core_env.properties):
- qlik.sense.url
- qlik.sense.redirect.url
- qlik.sense.enginePort
- qlik.sense.proxyPort
- qlik.sense.active.directory.name
- podium.qlik.username
- qlik.sense.skip.ticket
- qlik.sense.invert.security.model
- podium.qlik.dataconnection.name
The following two properties were also removed and have been superseded by pre-existing Qlik Sense Connector fields:
- qlik.sense.root.admin.directory.name (superseded by Admin User Directory)
- qlik.sense.root.admin.user.name (superseded by Admin User Name)
The following global settings remain in the core_env.properties file -- please see the comments there:
- is.publish.to.qlik.enabled
- podium.qlik.dataconnection.managed.entity.generating.from
- podium.qlik.dataconnection.addressed.registered.entity.generating.from
- podium.qlik.replace.previous.script
- qlik.sense.data.deletion.deferral.period.hours
- podium.qlik.dataconnection.sql.openquote.char
- podium.qlik.dataconnection.sql.closequote.char
- podium.qlik.appcreation.nodecommand
- podium.qlik.qsockclient.script.directory
- qlik.logs.purging.threshold
- qlik.sense.record.count.threshold.for.data.load
February 2022 SR1
CHANGE: Enable Catalog Auto-Creation of Sense Security Rules & Auto Add Newly Discovered QVDs to Local/AD Groups
The property controlling these capabilities is now named "qlik.sense.invert.security.model". It was formerly named "qlik.sense.auto.create.security.rules" when introduced in the initial February 2022 release. See earlier description of QDCB-1114.
# Normally, on QVD import, Catalog creates Groups that shadow Sense Data Connections. QVD entities are then added to
# these Groups. A Sense admin creates Security Rules granting Users access to Data Connections. When a User logs
# in to Catalog, a security audit is conducted against Sense, and the User is added to Groups if they have access to the
# corresponding Data Connections.
#
# This model may be inverted. A Catalog admin can instead manually add QVD entities to Catalog local/AD groups. When
# this occurs, users running Publish to Qlik Sense may need Data Connection Security Rules created in order to load
# published data in Sense. Catalog can be configured to automatically create any needed Security Rules by setting this
# property to true. In addition, this property ensures that once a single QVD entity has been manually added to a Group,
# future QVD entities discovered during import in the same Data Connection folder will be automatically added to the
# same group.
#
# Properties 'qlik.sense.root.admin.directory.name' and 'qlik.sense.root.admin.user.name' must also be set.
# Formerly: qlik.sense.auto.create.security.rules. Default: false
#qlik.sense.invert.security.model=true
February 2022
ADDITION: Enable Catalog Auto-Creation of Sense Security Rules
Set this property to true to have Catalog auto-create Qlik Sense Data Connection Security Rules (if needed) as part of the Publish to Qlik Sense process. See earlier description.
# If Catalog local or AD Groups have had QVD Sources/Entities added to them, users running Publish to Qlik Sense
# may need Data Connection Security Rules created in order to load data in Sense. Normally, these Security Rules
# should be created and managed in Sense QMC. However, Catalog can be configured to automatically create any
# needed Security Rules if this property is set to true. Properties 'qlik.sense.root.admin.directory.name' and
# 'qlik.sense.root.admin.user.name' must also be set. Default: false
#qlik.sense.auto.create.security.rules=true
ADDITION: Alter SAML Identity Provider User Domain Name
Specify this property to set or alter the domain name of the user sent by the SAML identity provider (IdP) to Catalog. See earlier description.
# If the IdP does not append a domain and one is needed, or you wish to change the domain, set this property. If the
# property is set to a value of "test.com", a SAML principal name of "jdoe" would become "jdoe@test.com"; a SAML
# principal name of "jdoe@other.com" would become "jdoe@test.com". The known use case for this property is to match
# principal (aka user) names coming from the IdP with the users imported from Active Directory. Default: not set
#saml.alternate.domain=
November 2021 SR2
No changes.
November 2021 SR1
CHANGE: Publish to Qlik Sense Enhancement
The Publish to Qlik Sense RootAdmin user is now more widely applied. Before, it was only used when multiple domains were specified in property "qlik.sense.active.directory.name". Now, it is used to ensure a known, valid Sense user is being used for Publish to Qlik Sense. See earlier description of QDCB-1007.
# Enter the directory and user name of a Sense 'RootAdmin' user.
# Used to validate that the domain user being used for Publish to Qlik Sense has previously logged into the
# Sense server. This prevents users known only to Catalog being inadvertently created in Sense.
# Mandatory if multiple directories were specified in property 'qlik.sense.active.directory.name'.
qlik.sense.root.admin.directory.name=AD
qlik.sense.root.admin.user.name=sense-service
ADDITION: Extended Support for Fields in Prepare Dataflows That Are Also Pig Reserved Words
Prepare dataflow jobs will fail if fields are named using unanticipated Apache Pig reserved words. Such words can now be configured. See earlier description of QDCB-1107.
# Entity fields used in Prepare Dataflows may also be Pig reserved words (e.g., STORE). Frequently used reserved words
# are correctly handled if they are field names. This property may be used to augment the set of known reserved words
# with unanticipated words. Words must be comma separated. Default: not used
#pig.reserved.words.additional=register,CASE
Downloads
May 2022 SR1 Catalog Application
May 2022 SR1 Catalog Installer
About Qlik
Qlik converts complex data landscapes into actionable insights, driving strategic business outcomes. Serving over 40,000 global customers, our portfolio provides advanced, enterprise-grade AI/ML, data integration, and analytics. Our AI/ML tools, both practical and scalable, lead to better decisions, faster. We excel in data integration and governance, offering comprehensive solutions that work with diverse data sources. Intuitive analytics from Qlik uncover hidden patterns, empowering teams to address complex challenges and seize new opportunities. As strategic partners, our platform-agnostic technology and expertise make our customers more competitive.