Qlik Catalog Release Notes - February 2022 Initial Release to Service Release 2

djenkins-qlik — Fri, 20 Dec 2024 20:35:39 GMT

Table of Contents

The following release notes cover the versions of Qlik Catalog released in February 2022.

What's new in Qlik Catalog February 2022 SR2

Noteworthy Newly Resolved Issues with this Release

QDCB-1139 - Address Spring Framework Remote Code Execution (RCE) Vulnerability

Noteworthy Enhancements from February 2022 Initial Release

Business Metadata Export
Public APIs for User and Group Security Management
Load Support for Additional Parquet Types
New SAML Property to Modify User Domain Name
Single-node Catalog Upgraded to Use log4j Version 2.17.1

Business Metadata Export

Catalog has long supported business metadata import. However, once edited in Catalog, a second import would potentially overwrite changes. Now, business metadata can also be exported. This enables "roundtrip" business metadata maintenance. Business metadata may be exported to a CSV file, modified in a tool like Microsoft Excel, and then re-imported into Catalog. Please see the online help for more detail.

Public APIs for User and Group Management

A new public Security API enables:

retrieval of Users and Groups
creation, update and deletion of Users and Groups
creation, update and deletion of associations between Users and Groups (with a Role)
retrieval of Roles
clearing of the User-to-Group association cache that is evaluated at User logon

Please visit the "live" documentation included with Catalog at Support / API Documentation, where a README and example shell script are also available:

Load Support for Additional Parquet Types

NOTE: Only available on single-node and CDP7 multi-node installations. Not supported on EMR multi-node installations.

Previously, Catalog only supported loading Parquet file-format primitive types. Support has been added for several common logical types (see https://github.com/apache/parquet-format/blob/master/LogicalTypes.md). The set of types are decimals encoded in a variety of formats as well as timestamps (precision and scale numbers below are examples):

int32 decimalFieldName1 (DECIMAL(4,2))
int64 decimalFieldName2 (DECIMAL(14,2))
fixed_len_byte_array(9) decimalFieldName3 (DECIMAL(20,2))
binary decimalFieldName4 (DECIMAL(20,2));
int64 timestampFieldName1 (TIMESTAMP(MILLIS, true));

In addition, Catalog now supports timestamps encoded in the legacy, deprecated int96 primitive type:

int96 timestampFieldName2

Auto-Create Qlik Sense Security Rules

When a user logs-in to Catalog, an audit is conducted against Qlik Sense to determine the Data Connections to which the user has access. In Catalog, the user is then automatically joined to the Groups that correspond to the Data Connections, thereby granting them access to QVDs covered by those Data Connections. In this model, Security Rules, granting read access to Data Connections, are created and maintained in Sense.

However, one customer requested that Catalog also serve as a security system of record. This is STRONGLY DISCOURAGED as security is then managed from two distinct products. This customer wanted to add QVD Sources/Entities to local or AD groups and have Catalog create any needed Sense Security Rules.

Set this property to true to have Catalog auto-create Qlik Sense Data Connection Security Rules (if needed) as part of the Publish to Qlik Sense process.

# If Catalog local or AD Groups have had QVD Sources/Entities added to them, users running Publish to Qlik Sense

# may need Data Connection Security Rules created in order to load data in Sense. Normally, these Security Rules

# should be created and managed in Sense QMC. However, Catalog can be configured to automatically create any

# needed Security Rules if this property is set to true. Properties 'qlik.sense.root.admin.directory.name' and

# 'qlik.sense.root.admin.user.name' must also be set. Default: false

#qlik.sense.auto.create.security.rules=true

New SAML Property to Modify User Domain Name

If the login name being passed by the SAML identity provider (IdP) does not include a domain, and a fully qualified login name is required to match previously imported/created users, core_env property "saml.alternate.domain" may be used. It can also be used to alter/replace the domain name. The new property:

# If the IdP does not append a domain and one is needed, or you wish to change the domain, set this property. If the

# property is set to a value of "test.com", a SAML principal name of "jdoe" would become "jdoe@test.com"; a SAML

# principal name of "jdoe@other.com" would become "jdoe@test.com". The known use case for this property is to match

# principal (aka user) names coming from the IdP with the users imported from Active Directory. Default: not set

#saml.alternate.domain=

No Longer Supported

With the August 2021 release, CDH and HDP Hadoop cluster environments are no longer supported (supported Hadoop cluster environments are AWS EMR and CDP Private Cloud 7)
With the May 2021 release, Microsoft Internet Explorer 11 is no longer supported and Workflow Scheduler was removed

Resolved Defects

Includes issues resolved in the February 2022 SR2, SR1 and initial releases.

February 2022 SR2 (4.13.2)

Address Spring Framework Remote Code Execution (RCE) Vulnerability

Jira ID: QDCB-1139

Catalog includes a third-party library, Spring Framework, that contained a remote code execution (RCE) vulnerability. Please review the following for more detail: https://spring.io/blog/2022/03/31/spring-framework-rce-early-announcement

This release upgrades Spring Framework to version 5.2.20, which addresses the vulnerability.

As an alternative to deploying this Service Release, the Java Development Kit (JDK) used by Catalog may be downgraded to JDK 8.

February 2022 SR1 (4.13.1)

Support for Altering Name and Extension of Published Files

Jira ID: QDCB-1063

Support has been added for altering the name and extension of published files from the default of "publish-r-00000", "publish-r-00001", etc. File names eligible for change include those beginning with "publish-m" and "publish-r", and those beginning with six digits (e.g., "000000_0", seen on multi-node installations).

This feature applies to file formats of PARQUET and TEXTFILE. Files using format QVD are already named using the entity name and have an extension of "qvd" – in this case, setting the properties will have no effect. In addition, this feature only applies to publish targets of type "file" (e.g., File on S3 Storage).

Two new properties may be added to a Publish Job definition: "filename.naming.custom.prop" and "extension.naming.custom.prop". In the following screenshot, they are set to "schemaData" and "txt". This produces a result of "schemaData-00000.txt".

Each property has a special value that will alter its behavior.

If "filename.naming.custom.prop" is set to "attr.entity.name", then the entity name will be used as the new name.

If "extension.naming.custom.prop" is set to "auto", then the extension is automatically determined when format TEXTFILE is chosen:

csv will be used for comma-delimited data
tsv will be used for tab-delimited data
txt will be used for all other TEXTFILE data

Note that files of format PARQUET already have a "parquet" extension – the extension property, if set, will be ignored.

Auto Add Newly Discovered QVDs to Local/AD Groups

Jira ID: QDCB-1114

Normally, on QVD import, Catalog creates Groups that shadow Sense Data Connections. QVD entities are then added to these Groups. A Sense admin creates Security Rules granting Users access to Data Connections. When a User logs in to Catalog, a security audit is conducted against Sense, and the User is added to Groups if they have access to the corresponding Data Connections.

This model may be inverted. A Catalog admin can instead manually add QVD entities to Catalog local/AD groups. When this occurs, users running Publish to Qlik Sense may need Data Connection Security Rules created to load published data in Sense. Catalog can be configured to automatically create any needed Security Rules by setting this property to true. In addition, this property ensures that once a single QVD entity has been manually added to a Group, future QVD entities newly discovered during import in the same Data Connection folder will be automatically added to the same group.

Use of this inverted security model is STRONGLY DISCOURAGED as security is then managed from two distinct products.

The core_env property controlling this capability is now named "qlik.sense.invert.security.model". It was formerly named "qlik.sense.auto.create.security.rules".

# Normally, on QVD import, Catalog creates Groups that shadow Sense Data Connections. QVD entities are then added to
# these Groups. A Sense admin creates Security Rules granting Users access to Data Connections. When a User logs

# in to Catalog, a security audit is conducted against Sense, and the User is added to Groups if they have access to the

# corresponding Data Connections.

# This model may be inverted. A Catalog admin can instead manually add QVD entities to Catalog local/AD groups. When

# this occurs, users running Publish to Qlik Sense may need Data Connection Security Rules created in order to load

# published data in Sense. Catalog can be configured to automatically create any needed Security Rules by setting this

# property to true. In addition, this property ensures that once a single QVD entity has been manually added to a Group,

# future QVD entities discovered during import in the same Data Connection folder will be automatically added to the

# same group.

# Properties 'qlik.sense.root.admin.directory.name' and 'qlik.sense.root.admin.user.name' must also be set.

# Formerly: qlik.sense.auto.create.security.rules. Default: false

#qlik.sense.invert.security.model=true

Fix: Publish to Power BI of Registered Entity Uses Sample Data

Jira ID: QDCB-1129

When a non-managed (e.g., registered) entity is selected for Publish to Power BI, it is "loaded on demand" (aka ad-hoc promotion). All good records should be published. However, only sample data was published. This issue has been corrected.

Allow ADMIN Users to Demote Themselves

Jira ID: QDCB-1074

A user may now demote themselves from ADMIN to a role without access to the Security module (e.g., ANALYST). If doing so leaves the user without an ADMIN role association to any group, a warning will be displayed and the user will be logged-out.

Fix: Publish using Redshift Target Fails If Header Included

Jira ID: QDCB-1132

A Publish job, using a Redshift target, failed if the checkbox "Header (include field names)" was selected. It failed with error: [Amazon](500310) Invalid operation: unrecognized configuration parameter "skip.header.line.count". This issue has been corrected.

February 2022 Initial Release (4.13)

Single-node Catalog has been upgraded to use log4j version 2.17.1.

Fix Prepare Dataflow Editor Handling of Single Quote Character

Jira ID: QDCB-1127

In the Prepare Dataflow editor, transform expressions like:

Obfuscate (field, 'Replace All With Null')

became:

Obfuscate (field,

when the expression was re-opened for further edit. A bug in the handling of the single quote character has been fixed.

Upgrade notes

Migrating to or Upgrading Tomcat 9

Beginning with the May 2021 release, only Apache Tomcat 9 is supported. The installer will prohibit other versions. If using Tomcat 7, please first initiate a migration to Tomcat 9 before installing this release. Then, when installing, the upgrade option (-u) is NOT used.

These instructions may also be used to upgrade from an older version of Tomcat 9 to a newer version.

Step	Sample Commands
Shutdown and rename old Tomcat 7 or 9	cd /usr/local/qdc (or cd /usr/local/podium) ./apache-tomcat-<OLD_VERSION>/bin/shutdown.sh mv apache-tomcat-<OLD_VERSION> old-apache-tomcat
Download and expand Tomcat 9 - NOTE: adjust version 9.0.56 to use latest 9.0.x series	wget https://archive.apache.org/dist/tomcat/tomcat-9/v9.0.56/bin/apache-tomcat-9.0.56.tar.gz tar -xf apache-tomcat-9.0.56.tar.gz rm apache-tomcat-9.0.56.tar.gz
Copy core_env.properties from old Tomcat to new Tomcat 9	cp old-apache-tomcat/conf/core_env.properties apache-tomcat-9.0.56/conf/
If migrating from Tomcat 7: Extract server.xml from podium.zip and copy to new Tomcat	unzip -j podium-4.<VERSION>-<BUILD>.zip podium/config/tomcat9-server.xml -d . mv ./tomcat9-server.xml apache-tomcat-9.0.56/conf/server.xml
If upgrading Tomcat 9: Copy server.xml from old Tomcat 9 to new Tomcat 9	cp old-apache-tomcat/conf/server.xml apache-tomcat-9.0.56/conf/ If the old Tomcat 9 was configured for HTTPS, and the keystore (jks file) was stored in the old Tomcat directory, migrate it to the new Tomcat directory, and update conf/server.xml to reference it. Consider placing the keystore file in a non-Tomcat directory such as /usr/local/qdc/keystore.
Configure QDCinstaller.properties for Tomcat 9	Whether using an existing QDCinstaller.properties file from a previous install, or configuring one for the first time, ensure that it is updated to point to Tomcat 9: TOMCAT_HOME=/usr/local/podium/apache-tomcat-9.0.56
Finally, when the installer is run, *do NOT* specify upgrade mode (-u)**, as some files should be created as if it were a first-time install.	./QDCinstaller.sh

At this point, Tomcat 9, if newly installed, will support only HTTP on port 8080.

Verify successful Qlik Catalog startup and basic functionality.

Additional configuration will be required to enable HTTPS on port 8443, apply security headers, etc. If Tomcat 7 used HTTPS, the keystore (jks file) containing the public-private keypair should be copied to Tomcat 9 and conf/server.xml updated.

In addition, Tomcat 7 may have been configured as a service. It should be disabled. Tomcat 9 may be configured as a service to automatically start.

Please see the install guide for guidance on both.

Process if Upgrading From June 2020 or Earlier

Do not attempt to upgrade until the following is understood

If upgrading from a version of Qlik Catalog prior to September 2020 (4.7) there are utilities that MUST be run after Catalog is upgraded. Once run, the utilities need never be run again.

The server may not start until the first two utilities have been run and will log a WARN at startup until the third is run. Do NOT upgrade the server until familiar with these utilities and the information required to run them. It will take time to gather this information. Gathering the information BEFORE Catalog is upgraded will minimize downtime.

Run the utilities in this order:

jwt2CertsUtility -- please review readme.txt

This will be required if Qlik Sense Connectors have been defined to load QVDs.

Will need to gather networking info and certificate files from Qlik Sense servers.

May be run from any directory.

singleNodeUpgradeForEntitiesWithBadOrUglyData.sh -- please review comment in script

This will be required only if the installation is single-node.

Will need podium_dist database info if defaults altered.

May be run from any directory.

singleNodeUpgradeToGrantReadOnlyUserAccessToDistSchemas.sh -- please review comment in script

This will be required only if the installation is single-node.

Will need podium_dist database info if defaults altered.

May be run from any directory.

Log of Changes to File core_env.properties

A chronological listing (most recent first) of additions, changes in behavior, and deletions to the primary global configuration file, core_env.properties.

February 2022 SR1

CHANGE: Enable Catalog Auto-Creation of Sense Security Rules & Auto Add Newly Discovered QVDs to Local/AD Groups

The property controlling these capabilities is now named "qlik.sense.invert.security.model". It was formerly named "qlik.sense.auto.create.security.rules" when introduced in the initial February 2022 release. See earlier description of QDCB-1114.

# in to Catalog, a security audit is conducted against Sense, and the User is added to Groups if they have access to the

# corresponding Data Connections.

# This model may be inverted. A Catalog admin can instead manually add QVD entities to Catalog local/AD groups. When

# this occurs, users running Publish to Qlik Sense may need Data Connection Security Rules created in order to load

# published data in Sense. Catalog can be configured to automatically create any needed Security Rules by setting this

# property to true. In addition, this property ensures that once a single QVD entity has been manually added to a Group,

# future QVD entities discovered during import in the same Data Connection folder will be automatically added to the

# same group.

# Properties 'qlik.sense.root.admin.directory.name' and 'qlik.sense.root.admin.user.name' must also be set.

# Formerly: qlik.sense.auto.create.security.rules. Default: false

#qlik.sense.invert.security.model=true

February 2022

ADDITION: Enable Catalog Auto-Creation of Sense Security Rules

Set this property to true to have Catalog auto-create Qlik Sense Data Connection Security Rules (if needed) as part of the Publish to Qlik Sense process. See earlier description.

# If Catalog local or AD Groups have had QVD Sources/Entities added to them, users running Publish to Qlik Sense

# may need Data Connection Security Rules created in order to load data in Sense. Normally, these Security Rules

# should be created and managed in Sense QMC. However, Catalog can be configured to automatically create any

# needed Security Rules if this property is set to true. Properties 'qlik.sense.root.admin.directory.name' and

# 'qlik.sense.root.admin.user.name' must also be set. Default: false

#qlik.sense.auto.create.security.rules=true

ADDITION: Alter SAML Identity Provider User Domain Name

Specify this property to set or alter the domain name of the user sent by the SAML identity provider (IdP) to Catalog. See earlier description.

# If the IdP does not append a domain and one is needed, or you wish to change the domain, set this property. If the

# property is set to a value of "test.com", a SAML principal name of "jdoe" would become "jdoe@test.com"; a SAML

# principal name of "jdoe@other.com" would become "jdoe@test.com". The known use case for this property is to match

# principal (aka user) names coming from the IdP with the users imported from Active Directory. Default: not set

#saml.alternate.domain=

November 2021 SR2

No changes.

November 2021 SR1

CHANGE: Publish to Qlik Sense Enhancement

The Publish to Qlik Sense RootAdmin user is now more widely applied. Before, it was only used when multiple domains were specified in property "qlik.sense.active.directory.name". Now, it is used to ensure a known, valid Sense user is being used for Publish to Qlik Sense. See earlier description of QDCB-1007.

# Enter the directory and user name of a Sense 'RootAdmin' user.

# Used to validate that the domain user being used for Publish to Qlik Sense has previously logged into the

# Sense server. This prevents users known only to Catalog being inadvertently created in Sense.

# Mandatory if multiple directories were specified in property 'qlik.sense.active.directory.name'.

qlik.sense.root.admin.directory.name=AD

qlik.sense.root.admin.user.name=sense-service

ADDITION: Extended Support for Fields in Prepare Dataflows That Are Also Pig Reserved Words

Prepare dataflow jobs will fail if fields are named using unanticipated Apache Pig reserved words. Such words can now be configured. See earlier description of QDCB-1107.

# Entity fields used in Prepare Dataflows may also be Pig reserved words (e.g., STORE). Frequently used reserved words

# are correctly handled if they are field names. This property may be used to augment the set of known reserved words

# with unanticipated words. Words must be comma separated. Default: not used

#pig.reserved.words.additional=register,CASE

Downloads

Qlik Catalog February 2022 SR2 - Application

Qlik Catalog February 2022 SR2 - Installer

About Qlik

Qlik converts complex data landscapes into actionable insights, driving strategic business outcomes. Serving over 40,000 global customers, our portfolio provides advanced, enterprise-grade AI/ML, data integration, and analytics. Our AI/ML tools, both practical and scalable, lead to better decisions, faster. We excel in data integration and governance, offering comprehensive solutions that work with diverse data sources. Intuitive analytics from Qlik uncover hidden patterns, empowering teams to address complex challenges and seize new opportunities. As strategic partners, our platform-agnostic technology and expertise make our customers more competitive.

qlik.com

article Qlik Catalog Release Notes - February 2022 Initial Release to Service Release 2 in Release Notes

Qlik Catalog Release Notes - February 2022 Initial Release to Service Release 2

What's new in Qlik Catalog February 2022 SR2

Noteworthy Enhancements from February 2022 Initial Release

Business Metadata Export

Public APIs for User and Group Management

Load Support for Additional Parquet Types

Resolved Defects

February 2022 SR2 (4.13.2)

Address Spring Framework Remote Code Execution (RCE) Vulnerability

February 2022 SR1 (4.13.1)

Support for Altering Name and Extension of Published Files

Auto Add Newly Discovered QVDs to Local/AD Groups

Fix: Publish to Power BI of Registered Entity Uses Sample Data

Allow ADMIN Users to Demote Themselves

Fix: Publish using Redshift Target Fails If Header Included

February 2022 Initial Release (4.13)

Fix Prepare Dataflow Editor Handling of Single Quote Character

Upgrade notes

Migrating to or Upgrading Tomcat 9

Process if Upgrading From June 2020 or Earlier

Log of Changes to File core_env.properties

February 2022 SR1

February 2022

November 2021 SR2

November 2021 SR1

Downloads