Table of Contents
- What's new in Qlik Catalog August 2022 Service Release 2
- Fixed: Business Metadata Import "Overwrite Tags" Removes Entity Properties
- What's new in Qlik Catalog August 2022 SR1
- Resolved Defects
- Business Metadata Import: Support Tag Overwrite
- Sub-Groups Incorrectly Inheriting Role from User-Group Assignment
- Incorrect Field Ordering for API Supporting Qlik Enterprise Manager (QEM)
- Enhance JDBC Test Connection Functionality
- Fail With Clear Error If Cannot Find core_env.properties
- Example API Script for Create Entity from FDL
- Upgrade notes
- Migrating to or Upgrading Tomcat 9
- Process if Upgrading From June 2020 or Earlier
- Log of Changes to File core_env.properties
- August 2022
- May 2022 SR1
- May 2022
- February 2022 SR1
- February 2022
- November 2021 SR2
- November 2021 SR1
- Downloads
The following release notes cover the versions of Qlik Catalog released in August 2022.
What's new in Qlik Catalog August 2022 Service Release 2
A single critical defect was fixed in Service Release 2.
Fixed: Business Metadata Import "Overwrite Tags" Removes Entity Properties
Jira ID: QDCB-1215
Also released in November 2022 SR2 and February 2023.
When importing business metadata, if the "Overwrite Tags" checkbox was selected, all entity properties were removed. As these properties primarily control how an entity is loaded (e.g., property "src.file.glob"), data loads failed.
This issue has been addressed: the "Overwrite Tags" checkbox may now be safely selected.
Further, for JDBC entities, the missing properties are restored when the entity is next loaded. This repair is dependent on the entity retaining the original name of the JDBC relational table (i.e., the entity cannot have been renamed). The repair occurs for JDBC entities that are missing the "src.file.glob" property.
New logging has been introduced to validate that the two facets of the fix are present:
(1) When importing business metadata with "Overwrite Tags" selected, the following line of logging will be seen twice for each entity:
2023-02-02 15:49:26,712 INFO Entity properties were attached prior to clearing tags [MetadataLoaderSvcImpl[https-jsse-nio-8443-exec-1]]
(2) When loading JDBC entities that require repair, the following line of logging will be seen once per entity being repaired:
2023-02-02 15:50:48,523 INFO Restoring entity properties for podium_core.pd_group [MetadataLoaderSvcImpl[Podium-LOAD-2-podium_core_1215.pd_group.20230202155029]]
What's new in Qlik Catalog August 2022 SR1
Noteworthy Enhancements in August 2022 Initial Release
- Ubuntu 22.04 (LTS) is now a supported Linux distribution
- Upgrade of all Qlik Catalog supplied container images
- Standard updates of vulnerable third-party dependencies, including Spring Framework 5.3.21
- Standard updates of Apache Tomcat and PostgreSQL for first-time installs using QDCprereqs.sh
No Longer Supported
- With the August 2022 release, field-level encryption has been removed.
- With the August 2021 release, CDH and HDP Hadoop cluster environments are no longer supported (supported Hadoop cluster environments are AWS EMR and CDP Private Cloud 7)
- With the May 2021 release, Microsoft Internet Explorer 11 is no longer supported and Workflow Scheduler was removed
Resolved Defects
QDC-1349 - Business Metadata Import: Support Tag Overwrite
QDCB-1177 - Sub-Groups Incorrectly Inheriting Role from User-Group Assignment
QDCB-1180 - Incorrect Field Ordering for API Supporting Qlik Enterprise Manager (QEM)
QDCB-1173 - Enhance JDBC Test Connection Functionality
QDCB-1178 - Fail With Clear Error If Cannot Find core_env.properties
QDCB-1179 - Example API Script for Create Entity from FDL
Business Metadata Import: Support Tag Overwrite
Jira ID: QDC-1349
Previously, business metadata import was "additive" for tags – tags specified in the spreadsheet (CSV or XLS) were added to the existing source/entity/field tag set. With this enhancement, the source/entity/field tag set can be reset (aka overwritten or cleared).
This can be done both for the entire business metadata import (the entire file) by selecting the “Overwrite Tags” UI checkbox, or on a row-by-row basis by inserting a column with header “Overwrite Tags” between “Technical Description” and “Tag1” columns and setting the value for any source/entity/field tag set to true/TRUE in that column. The source/entity/field tag set will then be overwritten upon metadata import. If no tags are supplied in the sheet, the tag set is effectively cleared.
The "Overwrite Tags" column does not have to be present. If the column is present, TRUE/true can be put in as
many or few cells as needed (for example: all, none or only one).
Sub-Groups Incorrectly Inheriting Role from User-Group Assignment
Jira ID: QDCB-1177
A security group can have sub-groups. The sub-groups should inherit the role the user has with the base group. The following inconsistencies have been addressed:
- The Prepare "Add Source" widget now shows sub-group entities if the user's base group role allows Source module modification (add and edit data source permissions)
- Publish job creation now accepts sub-group entities if the user's base group role allows publish configuration
- The Prepare "Add Source" widget and "Prepare from Cart" now consistently enforce which sub-group entities can be added to a Prepare Dataflow
Incorrect Field Ordering for API Supporting Qlik Enterprise Manager (QEM)
Jira ID: QDCB-1180
The Catalog code that was processing the QEM "entity add" API call was incorrectly generating field indexes. When loading data from file format (for example, from S3), the data was mapped to incorrect columns. The indexes are now correctly set.
Enhance JDBC Test Connection Functionality
Jira ID: QDCB-1173
When testing a JDBC Source Connection or Publish Target, the root cause error was not being captured and returned to the user. The underlying JDBC driver error is now returned to the user (e.g., connection timed-out or incorrect username/password) – for example:
The connection could not be verified.
core.error.code.TEST_CONNECTION_WITH_MESSAGE – Test database connection attempt failed, please check the connection string, user name and password: PSQLException:
FATAL: password authentication failed for user "podium_mdxxx"
Fail With Clear Error If Cannot Find core_env.properties
Jira ID: QDCB-1178
If the primary configuration file core_env.properties was missing, or Tomcat's bin/setenv.sh was itself missing or incorrect, Catalog was defaulting to an embedded version of the file, failing to startup, and outputting a misleading error message. Now, when this situation occurs, a very clear error message is displayed in catalina.out:
Caused by: java.lang.RuntimeException: Could not find core_env.properties, Java system property podium.conf.dir should be set in bin/setenv.sh
Example API Script for Create Entity from FDL
Jira ID: QDCB-1179
An entity and its fields can be created using an FDL (File Description Language) specification. A shell script, FDL file and sample data are now included with the API Documentation. This example also illustrates how to script a multipart form upload.
Upgrade notes
Migrating to or Upgrading Tomcat 9
Beginning with the May 2021 release, only Apache Tomcat 9 is supported. The installer will prohibit other versions. If using Tomcat 7, please first initiate a migration to Tomcat 9 before installing this release. Then, when installing, the upgrade option (-u) is NOT used.
These instructions may also be used to upgrade from an older version of Tomcat 9 to a newer version.
|
Step |
Sample Commands |
|
Shutdown and rename old Tomcat 7 or 9 |
cd /usr/local/qdc (or cd /usr/local/podium) ./apache-tomcat-<OLD_VERSION>/bin/shutdown.sh mv apache-tomcat-<OLD_VERSION> old-apache-tomcat |
|
Download and expand Tomcat 9 - NOTE: adjust version 9.0.64 to use latest 9.0.x series |
wget https://archive.apache.org/dist/tomcat/tomcat-9/v9.0.64/bin/apache-tomcat-9.0.64.tar.gz tar -xf apache-tomcat-9.0.64.tar.gz rm apache-tomcat-9.0.64.tar.gz |
|
Copy core_env.properties from old Tomcat to new Tomcat 9 |
cp old-apache-tomcat/conf/core_env.properties apache-tomcat-9.0.64/conf/ |
|
If migrating from Tomcat 7: Extract server.xml from podium.zip and copy to new Tomcat |
unzip -j podium-4.<VERSION>-<BUILD>.zip podium/config/tomcat9-server.xml -d . mv ./tomcat9-server.xml apache-tomcat-9.0.64/conf/server.xml |
|
If upgrading Tomcat 9: Copy server.xml from old Tomcat 9 to new Tomcat 9 |
cp old-apache-tomcat/conf/server.xml apache-tomcat-9.0.64/conf/ If the old Tomcat 9 was configured for HTTPS, and the keystore (jks file) was stored in the old Tomcat directory, migrate it to the new Tomcat directory, and update conf/server.xml to reference it. Consider placing the keystore file in a non-Tomcat directory such as /usr/local/qdc/keystore. |
|
Configure QDCinstaller.properties for Tomcat 9 |
Whether using an existing QDCinstaller.properties file from a previous install, or configuring one for the first time, ensure that it is updated to point to Tomcat 9: TOMCAT_HOME=/usr/local/podium/apache-tomcat-9.0.64 |
|
Finally, when the installer is run, do NOT specify upgrade mode (-u), as some files should be created as if it were a first-time install. |
./QDCinstaller.sh |
At this point, Tomcat 9, if newly installed, will support only HTTP on port 8080.
Verify successful Qlik Catalog startup and basic functionality.
Additional configuration will be required to enable HTTPS on port 8443, apply security headers, etc. If Tomcat 7 used HTTPS, the keystore (jks file) containing the public-private keypair should be copied to Tomcat 9 and conf/server.xml updated.
In addition, Tomcat 7 may have been configured as a service. It should be disabled. Tomcat 9 may be configured as a service to automatically start.
Please see the install guide for guidance on both.
Process if Upgrading From June 2020 or Earlier
Do not attempt to upgrade until the following is understood.
If upgrading from a version of Qlik Catalog prior to September 2020 (4.7) there are utilities that MUST be run after Catalog is upgraded. Once run, the utilities need never be run again.
The server may not start until the first two utilities have been run and will log a WARN at startup until the third is run. Do NOT upgrade the server until familiar with these utilities and the information required to run them. It will take time to gather this information. Gathering the information BEFORE Catalog is upgraded will minimize downtime.
Run the utilities in this order:
- jwt2CertsUtility -- please review readme.txt
This will be required if Qlik Sense Connectors have been defined to load QVDs.
Will need to gather networking info and certificate files from Qlik Sense servers.
May be run from any directory.
- singleNodeUpgradeForEntitiesWithBadOrUglyData.sh -- please review comment in script
This will be required only if the installation is single node.
Will need podium_dist database info if defaults altered.
May be run from any directory.
- singleNodeUpgradeToGrantReadOnlyUserAccessToDistSchemas.sh -- please review comment in script
This will be required only if the installation is single node.
Will need podium_dist database info if defaults altered.
May be run from any directory.
Log of Changes to File core_env.properties
A chronological listing (most recent first) of additions, changes in behavior, and deletions to the primary global configuration file, core_env.properties.
August 2022
ADDITION: Configurable LDAP Username Attribute for Active Directory Sync
When performing Active Directory (AD) sync using LDAP, Catalog hardcoded the use of LDAP user attribute sAMAccountName to extract the Catalog username. The attribute is now configurable, enabling a customer to choose an alternate attribute such as userPrincipalName. A new core_env property has been added:
# LDAP attribute used to extract the username from the LDAP record. Only
# the username before any "@domain" will be used. userPrincipalName is a
# logical alternative. Default: sAMAccountName
#ad.ldap.username.attribute=userPrincipalName
DELETION: Field-Level Encryption Removed
The ability to encrypt a field has been removed -- the "Encrypt" checkbox will no longer be present on the Field Information dialog. The following properties can now be retired (either commented-off or removed from core_env.properties):
- field.encryption.seed.reference
- field.encryption.cipher
May 2022 SR1
ADDITION: Fixed "Host" Header Poisoning
The Host header value should not automatically be trusted by Catalog. Instead, a protected configuration setting is introduced. See earlier description of QDCB-404.
# To protect against HTTP headers such as "Host" and "X-Forwarded-Host" being abused by a malicious client, set this
# to the hostname used by clients. It is embedded in link results returned by Catalog searches (e.g., /catalog/search).
# Restart required. Default: not set.
external.hostname=<FULLY_QUALIFIED_DOMAIN_NAME>
May 2022
CHANGE: Enable Publish to Qlik Sense by Default
Publish to Qlik Sense will now be enabled by default.
# Setting this to true will start showing 'Publish to Qlik' option in Podium UI cart checkouts.
# Default: true
is.publish.to.qlik.enabled=true
DELETION: Publish to Qlik Sense: Multiple Sense Server Support in User Interface
Configuration of specific Sense servers for Publish to Qlik Sense has been moved to the Catalog user interface. General settings remain in core_env.properties.
The following is taken from the "Qlik Catalog + Qlik Sense Integration Guide". The nine, new UI fields for Publish to Qlik Sense were formerly in core_env.properties. The following properties can now be retired (either commented-off or removed from core_env.properties):
- qlik.sense.url
- qlik.sense.redirect.url
- qlik.sense.enginePort
- qlik.sense.proxyPort
- qlik.sense.active.directory.name
- podium.qlik.username
- qlik.sense.skip.ticket
- qlik.sense.invert.security.model
- podium.qlik.dataconnection.name
The following two properties were also removed and have been superseded by pre-existing Qlik Sense Connector fields:
- qlik.sense.root.admin.directory.name (superseded by Admin User Directory)
- qlik.sense.root.admin.user.name (superseded by Admin User Name)
The following global settings remain in the core_env.properties file -- please see the comments there:
- is.publish.to.qlik.enabled
- podium.qlik.dataconnection.managed.entity.generating.from
- podium.qlik.dataconnection.addressed.registered.entity.generating.from
- podium.qlik.replace.previous.script
- qlik.sense.data.deletion.deferral.period.hours
- podium.qlik.dataconnection.sql.openquote.char
- podium.qlik.dataconnection.sql.closequote.char
- podium.qlik.appcreation.nodecommand
- podium.qlik.qsockclient.script.directory
- qlik.logs.purging.threshold
- qlik.sense.record.count.threshold.for.data.load
February 2022 SR1
CHANGE: Enable Catalog Auto-Creation of Sense Security Rules & Auto Add Newly Discovered QVDs to Local/AD Groups
The property controlling these capabilities is now named "qlik.sense.invert.security.model". It was formerly named "qlik.sense.auto.create.security.rules" when introduced in the initial February 2022 release. See earlier description of QDCB-1114.
# Normally, on QVD import, Catalog creates Groups that shadow Sense Data Connections. QVD entities are then added to
# these Groups. A Sense admin creates Security Rules granting Users access to Data Connections. When a User logs
# in to Catalog, a security audit is conducted against Sense, and the User is added to Groups if they have access to the
# corresponding Data Connections.
#
# This model may be inverted. A Catalog admin can instead manually add QVD entities to Catalog local/AD groups. When
# this occurs, users running Publish to Qlik Sense may need Data Connection Security Rules created in order to load
# published data in Sense. Catalog can be configured to automatically create any needed Security Rules by setting this
# property to true. In addition, this property ensures that once a single QVD entity has been manually added to a Group,
# future QVD entities discovered during import in the same Data Connection folder will be automatically added to the
# same group.
#
# Properties 'qlik.sense.root.admin.directory.name' and 'qlik.sense.root.admin.user.name' must also be set.
# Formerly: qlik.sense.auto.create.security.rules. Default: false
#qlik.sense.invert.security.model=true
February 2022
ADDITION: Enable Qlik Catalog Auto-Creation of Sense Security Rules
Set this property to true to have Catalog auto-create Qlik Sense Data Connection Security Rules (if needed) as part of the Publish to Qlik Sense process. See earlier description.
# If Catalog local or AD Groups have had QVD Sources/Entities added to them, users running Publish to Qlik Sense
# may need Data Connection Security Rules created in order to load data in Sense. Normally, these Security Rules
# should be created and managed in Sense QMC. However, Catalog can be configured to automatically create any
# needed Security Rules if this property is set to true. Properties 'qlik.sense.root.admin.directory.name' and
# 'qlik.sense.root.admin.user.name' must also be set. Default: false
#qlik.sense.auto.create.security.rules=true
ADDITION: Alter SAML Identity Provider User Domain Name
Specify this property to set or alter the domain name of the user sent by the SAML identity provider (IdP) to Catalog. See earlier description.
# If the IdP does not append a domain and one is needed, or you wish to change the domain, set this property. If the
# property is set to a value of "test.com", a SAML principal name of "jdoe" would become "jdoe@test.com"; a SAML
# principal name of "jdoe@other.com" would become "jdoe@test.com". The known use case for this property is to match
# principal (aka user) names coming from the IdP with the users imported from Active Directory. Default: not set
#saml.alternate.domain=
November 2021 SR2
No changes.
November 2021 SR1
CHANGE: Publish to Qlik Sense Enhancement
The Publish to Qlik Sense RootAdmin user is now more widely applied. Before, it was only used when multiple domains were specified in property "qlik.sense.active.directory.name". Now, it is used to ensure a known, valid Sense user is being used for Publish to Qlik Sense. See earlier description of QDCB-1007.
# Enter the directory and user name of a Sense 'RootAdmin' user.
# Used to validate that the domain user being used for Publish to Qlik Sense has previously logged into the
# Sense server. This prevents users known only to Catalog being inadvertently created in Sense.
# Mandatory if multiple directories were specified in property 'qlik.sense.active.directory.name'.
qlik.sense.root.admin.directory.name=AD
qlik.sense.root.admin.user.name=sense-service
ADDITION: Extended Support for Fields in Prepare Dataflows that are also Pig Reserved Words
Prepare dataflow jobs will fail if fields are named using unanticipated Apache Pig reserved words. Such words can now be configured. See earlier description of QDCB-1107.
# Entity fields used in Prepare Dataflows may also be Pig reserved words (e.g., STORE). Frequently used reserved words
# are correctly handled if they are field names. This property may be used to augment the set of known reserved words
# with unanticipated words. Words must be comma separated. Default: not used
#pig.reserved.words.additional=register,CASE
Downloads
Qlik Catalog August 2022 SR1 - Application
Qlik Catalog August 2022 SR1 - Installer
About Qlik
Qlik converts complex data landscapes into actionable insights, driving strategic business outcomes. Serving over 40,000 global customers, our portfolio provides advanced, enterprise-grade AI/ML, data integration, and analytics. Our AI/ML tools, both practical and scalable, lead to better decisions, faster. We excel in data integration and governance, offering comprehensive solutions that work with diverse data sources. Intuitive analytics from Qlik uncover hidden patterns, empowering teams to address complex challenges and seize new opportunities. As strategic partners, our platform-agnostic technology and expertise make our customers more competitive.
