Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.
Table of Contents
The following release notes cover the versions of Qlik Catalog released in November 2022. Resolved defects and limitations for Qlik Catalog are also listed.
A single critical defect was fixed in Service Release 2.
Jira ID: QDCB-1215
Also released in August 2022 SR2 and February 2023.
When importing business metadata, if the "Overwrite Tags" checkbox was selected, all entity properties were removed. As these properties primarily control how an entity is loaded (e.g., property "src.file.glob"), data loads failed.
This issue has been addressed: the "Overwrite Tags" checkbox may now be safely selected.
Further, for JDBC entities, the missing properties are restored when the entity is next loaded. This repair is dependent on the entity retaining the original name of the JDBC relational table (i.e., the entity cannot have been renamed). The repair occurs for JDBC entities that are missing the "src.file.glob" property.
New logging has been introduced to validate that the two facets of the fix are present:
(1) When importing business metadata with "Overwrite Tags" selected, the following line of logging will be seen twice for each entity:
2023-02-02 15:49:26,712 INFO Entity properties were attached prior to clearing tags [MetadataLoaderSvcImpl[https-jsse-nio-8443-exec-1]]
(2) When loading JDBC entities that require repair, the following line of logging will be seen once per entity being repaired:
2023-02-02 15:50:48,523 INFO Restoring entity properties for podium_core.pd_group [MetadataLoaderSvcImpl[Podium-LOAD-2-podium_core_1215.pd_group.20230202155029]]
Update Publish Target Credentials Via UI and API
Allow Properties to be Added to Discover Module QVD Entities
Catalog Failed to Start When "en_US" Locale Was Not Used
Enhanced User Session Logging
Prevent QVD Sources from Being Orphaned
API to Create JDBC Entity
Required Configuration for RHEL 9, SELinux and podman
Includes issues resolved in the November 2022 initial release - November 2022 SR1.
Jira ID: QDC-1377
Publish target credentials are now editable in the UI. Previously, editing of a publish target was disabled if the target was in use by a publish job.
In addition, an API was added to update target credentials, /qdc/target/v1/targets/{id}/credentials
Details of this API are available in Qlik Catalog under Support / API Documentation, where the API may be tried out and a sample script using curl is available.
Jira ID: QDCB-1199
Previously, properties could not be added to Discover module QVD entities. This prevented the user from adding custom properties, as well as setting relevant properties (e.g., podium.qlik.script).
It is now possible to add properties to Discover module QVD entities. Caution should be used as most system properties should not be added to, or modified on, QVD entities.
Jira ID: QDCB-1195
Previously, Qlik Catalog did not start when installed on a host where the default locale was not "en_US" (English, United States), or if Java was configured or installed to use a locale other than "en_US".
Catalog now successfully starts with non "en_US" locales. Note: Qlik Catalog is not localized and all messages and formatting continue to use the "en_US" locale.
Jira ID: QDC-1376
Qlik Catalog now logs (in catalina.out) enhanced user session logging.
User login, logout (or session expiration) and duration are now logged:
2022-11-21 12:39:35,463 INFO User successfully authenticated: podium [SessionListenerWithMetrics[https-jsse-nio-8443-exec-2]]
2022-11-21 13:40:23,390 INFO User session invalidated: podium, duration: 1:00:47 [SessionListenerWithMetrics[Catalina-utility-1]]
In addition, two periodic counts of logins (current and total) since Tomcat restart are output:
2022-11-28 15:32:05,878 INFO type=COUNTER, name=current.logins.since.restart, count=1 [SessionListenerWithMetrics[https-jsse-nio-8443-exec-6]]
2022-11-28 15:32:05,878 INFO type=COUNTER, name=total.logins.since.restart, count=3 [SessionListenerWithMetrics[https-jsse-nio-8443-exec-6]]
Note: if Tomcat is restarted there will be "uncounted" unexpired sessions carried over from the prior execution.
Jira ID: QDCB-1200
Previously, if a QVD Path deletion (red dot) was accepted while a QVD sync or load was running, the associated source was not deleted, became disconnected from the Catalog data model, and could not be deleted. This then interfered with subsequent acceptance of the same QVD Path when the Qlik Sense Data Connection was again tagged.
The deletion of a QVD Path is now rejected if a QVD sync or load is running.
Jira ID: QDCB-1184
An API was added to script the creation of JDBC entities. This is an alternative to the standard UI wizard for adding relational schema, table and column metadata via JDBC. To use this API, a JDBC source connection must already exist, as well as a source that uses it (requiring one entity to be added via the UI wizard). Once these exist, API /qdc/entity/v2/createFromFlatFile may be used.
Details of this API are available in Catalog under Support / API Documentation, where the API may be tried out and a sample script using curl is available
If Qlik Catalog is installed on Red Hat Enterprise Linux (RHEL) 9.1+ (and some versions of 😎 and SELinux is enabled (the default on RHEL 9), configuration changes must be made to successfully use the podman containers included by Qlik Catalog to import (load) and export (publish) QVDs, and to load XML data. NOTE: The alternative to the below procedure is to disable SELinux.
For QVD import and export, do the following, which will create a dedicated "temp" directory and configure SELinux and Catalog to use it. This temporary directory is where the engine container writes its output files. It must be in the root directory as many of the other standard locations (e.g., /tmp and /usr/local) have been black-listed.
As root:
# mkdir /tmp-catalog
# chown qdc:qdc /tmp-catalog
# semanage fcontext -a -s system_u -t container_file_t /tmp-catalog
# restorecon -F -v /tmp-catalog
As the Qlik Catalog service account user (e.g., qdc), edit core_env.properties, and find and set the following property:
qvd.conversion.temp.directory=/tmp-catalog
Also, as the Qlik Catalog service account, edit /usr/local/qdc/qlikcore/pod_qlikContainers.yml and uncomment the following two blocks:
# Uncomment the following volumeMount.
- mountPath: /tmp-catalog
name: tmp-catalog
and:
# Uncomment the following volume.
- hostPath:
path: /tmp-catalog
type: Directory
name: tmp-catalog
To ensure the podman containers are started on host reboot, the container start must be made dependent on the user service for the service account user (e.g., qdc).
As root, edit /etc/systemd/system/qlikContainers.service and follow the comment:
#After=syslog.target network.target qdc_pg.service
# A dependency on the service account user service is required for RHEL 9 with SELinux.
# Please see the November 2022 SR1 release notes or the February 2023 installation guide.
# Uncomment the below Requires and After; comment off the above Requires and After.
Requires=user@1000.service qdc_pg.service
After=user@1000.service qdc_pg.service
You can ensure user "qdc" has the correct ID by running "id -u qdc". If the correct numeric value is not used in the service file, update the values above.
Then, uncomment similar lines in file /etc/systemd/system/nextgen-xml.service.
When all changes have been made, run "systemctl reboot" to restart the system.
No noteworthy issues were resolved. Please see noteworthy features below.
This section pertains to Catalog installations that use podman and are having issues starting and stopping containers. Please also apply this command before upgrading. Podman is most likely in-use on RHEL 8, and Ubuntu 20.04 and 22.04, installations. The following command must be run once:
sudo loginctl enable-linger qdc
On most installs, it will be the root user that must run this. Replace "qdc" if a different Linux service account is used. Then, reboot the server (sudo shutdown -r now).
This advice assumes that the containers have been configured to run as services, either manually or by using QDCpostinstall.sh (now CatalogPostInstall.sh). "enable-linger" allows services running as a non-root user (i.e., qdc) to continue running cleanly without the user being logged-in. It is needed to cleanly shutdown and restart services using commands such as launch_qlikContainers.sh and shutdown_qlikContainers.sh.
Without this, errors will be seen when shutting-down pods and containers, particularly important when Catalog upgrade is run. Here is the error seen at upgrade:
Info: Qlik Licenses container running - shutting down & removing
error opening file `/run/user/1001/crun/88febfa150a1c8d207cfe1cc82f387c8f27d792f2d37fd0276a7858fc6255140/status`: No such file or directory
Error: timed out waiting for file /tmp/podman-run-1001/libpod/tmp/exits/88febfa150a1c8d207cfe1cc82f387c8f27d792f2d37fd0276a7858fc6255140: internal libpod error
Beginning with the November 2022 release, the installer will enable linger automatically on initial install if either CatalogFirstTime.sh or CatalogPrereqs.sh is used.
The following command can be used to verify linger status (can be run while logged-in as qdc):
loginctl user-status qdc | grep inger
Noteworthy changes and features.
For JDBC loads, customers may need to load only those rows introduced into the source database table since the last load into Qlik Catalog. A new entity property, src.table.new.records.filter.column, allows a customer to specify a table column that is used to identify new rows. Qlik Catalog will automatically build a WHERE-clause and append it to the SQL SELECT statement specified in property src.file.glob.
Logically, this column should be an ever-increasing numeric ID column, an IDENTITY column, found in many relational systems. A timestamp column like creation_ttz could be used if it was guaranteed to always increase and be unique—duplicate timestamps would be problematic.
Qlik Catalog uses the filter column to build a distribution database profile table query to find the maximum value of the column. The resulting value is used to automatically append a WHERE clause to the src.file.glob value which is used to retrieve data from the source table
Each time a load is executed this process is repeated; thereby ensuring only new rows are read from the source database table. To ensure consistent load behavior, the property src.table.new.records.filter.column should be set immediately after the entity is created and no later than before the second load is run.
Please see the online help for more details and a video demonstration.
Publish Job: Use Business Name for Header
The Publish Job "Entities" File Format dialog has been modified to include a new checkbox "Use Business Name for Header" – it should be "checkable" whenever the "Header" checkbox is checkable, generally TEXTFILE formats being output to some sort of file (e.g., LOCALFILE and S3; but not Amazon or Hadoop or QCS_S3):
Originally released as part of August 2022 SR1.
Previously, business metadata import was "additive" for tags – tags specified in the spreadsheet (CSV or XLS) were added to the existing source/entity/field tag set. With this enhancement, the source/entity/field tag set can be reset (aka overwritten or cleared).
This can be done both for the entire business metadata import (the entire file) by selecting the “Overwrite Tags” UI checkbox, or on a row-by-row basis by inserting a column with header “Overwrite Tags” between “Technical Description” and “Tag1” columns and setting the value for any source/entity/field tag set to true/TRUE in that column. The source/entity/field tag set will then be overwritten upon metadata import. If no tags are supplied in the sheet, the tag set is effectively cleared.
The "Overwrite Tags" column does not have to be present. If the column is present, TRUE/true can be put in as many or few cells as needed (for example: all, none or only one).
Catalog now supports the concurrent use of both SAML SSO and PODIUM (username and password) authentication. This enables end-users to logon using SSO while at the same time supporting API usage, including Sense and QEM API usage, that requires traditional username and password authentication.
Previously two Tomcat instances were required. This can now be accomplished using only a single Tomcat instance. Please see the install guide for details.
The entire Qlik Catalog first time installation process can now be driven by a single script, CatalogFirstTime.sh. This script, run as a sudo capable user, automatically:
The above three Qlik Catalog installation steps may still be separately run using:
All the above scripts require properties to be set in CatalogInstaller.properties (formerly QDCinstaller.properties). Upgrades continue to use the main installer script, "./CatalogInstaller.sh -u".
Please see the install guide for more details.
NOTE: on first time install, initial password properties must be set in CatalogInstaller.properties for the two accounts to be created by the installer:
NOTE: the installer now quickly detects corrupt podium zip files (e.g., podium-4.15-16448_Linux_X64.zip) and fails the installation process at the beginning.
Beginning with the May 2021 release, only Apache Tomcat 9 is supported. The installer will prohibit other versions. If using Tomcat 7, please first initiate a migration to Tomcat 9 before installing this release. Then, when installing, the upgrade option (-u) is NOT used.
These instructions may also be used to upgrade from an older version of Tomcat 9 to a newer version.
Step |
Sample Commands |
Shutdown and rename old Tomcat 7 or 9 |
cd /usr/local/qdc (or cd /usr/local/podium) ./apache-tomcat-<OLD_VERSION>/bin/shutdown.sh mv apache-tomcat-<OLD_VERSION> old-apache-tomcat |
Download and expand Tomcat 9 - NOTE: adjust version 9.0.64 to use latest 9.0.x series |
wget https://archive.apache.org/dist/tomcat/tomcat-9/v9.0.64/bin/apache-tomcat-9.0.64.tar.gz tar -xf apache-tomcat-9.0.64.tar.gz rm apache-tomcat-9.0.64.tar.gz |
Copy core_env.properties from old Tomcat to new Tomcat 9 |
cp old-apache-tomcat/conf/core_env.properties apache-tomcat-9.0.64/conf/ |
If migrating from Tomcat 7: Extract server.xml from podium.zip and copy to new Tomcat |
unzip -j podium-4.<VERSION>-<BUILD>.zip podium/config/tomcat9-server.xml -d . mv ./tomcat9-server.xml apache-tomcat-9.0.64/conf/server.xml |
If upgrading Tomcat 9: Copy server.xml from old Tomcat 9 to new Tomcat 9 |
cp old-apache-tomcat/conf/server.xml apache-tomcat-9.0.64/conf/ If the old Tomcat 9 was configured for HTTPS, and the keystore (jks file) was stored in the old Tomcat directory, migrate it to the new Tomcat directory, and update conf/server.xml to reference it. Consider placing the keystore file in a non-Tomcat directory such as /usr/local/qdc/keystore. |
Configure QDCinstaller.properties for Tomcat 9 |
Whether using an existing QDCinstaller.properties file from a previous install, or configuring one for the first time, ensure that it is updated to point to Tomcat 9: TOMCAT_HOME=/usr/local/podium/apache-tomcat-9.0.64 |
Finally, when the installer is run, do NOT specify upgrade mode (-u), as some files should be created as if it were a first-time install. |
./QDCinstaller.sh |
At this point, Tomcat 9, if newly installed, will support only HTTP on port 8080.
Verify successful Qlik Catalog startup and basic functionality.
Additional configuration will be required to enable HTTPS on port 8443, apply security headers, etc. If Tomcat 7 used HTTPS, the keystore (jks file) containing the public-private keypair should be copied to Tomcat 9 and conf/server.xml updated.
In addition, Tomcat 7 may have been configured as a service. It should be disabled. Tomcat 9 may be configured as a service to automatically start.
Please see the install guide for guidance on both.
Do not attempt to upgrade until the following is understood.
If upgrading from a version of Qlik Catalog prior to September 2020 (4.7) there are utilities that MUST be run after Catalog is upgraded. Once run, the utilities need never be run again.
The server may not start until the first two utilities have been run and will log a WARN at startup until the third is run. Do NOT upgrade the server until familiar with these utilities and the information required to run them. It will take time to gather this information. Gathering the information BEFORE Catalog is upgraded will minimize downtime.
Run the utilities in this order:
This will be required if Qlik Sense Connectors have been defined to load QVDs.
Will need to gather networking info and certificate files from Qlik Sense servers.
May be run from any directory.
This will be required only if the installation is single node.
Will need podium_dist database info if defaults altered.
May be run from any directory.
This will be required only if the installation is single node.
Will need podium_dist database info if defaults altered.
May be run from any directory.
A chronological listing (most recent first) of additions, changes in behavior, and deletions to the primary global configuration file, core_env.properties.
No additions, changes or deletions.
ADDITION: Configurable LDAP Username Attribute for Active Directory Sync
When performing Active Directory (AD) sync using LDAP, Catalog hardcoded the use of LDAP user attribute sAMAccountName to extract the Catalog username. The attribute is now configurable, enabling a customer to choose an alternate attribute such as userPrincipalName. A new core_env property has been added:
# LDAP attribute used to extract the username from the LDAP record. Only
# the username before any "@domain" will be used. userPrincipalName is a
# logical alternative. Default: sAMAccountName
#ad.ldap.username.attribute=userPrincipalName
DELETION: Field-Level Encryption Removed
The ability to encrypt a field has been removed -- the "Encrypt" checkbox will no longer be present on the Field Information dialog. The following properties can now be retired (either commented-off or removed from core_env.properties):
ADDITION: Fixed "Host" Header Poisoning
The Host header value should not automatically be trusted by Catalog. Instead, a protected configuration setting is introduced. See earlier description of QDCB-404.
# To protect against HTTP headers such as "Host" and "X-Forwarded-Host" being abused by a malicious client, set this
# to the hostname used by clients. It is embedded in link results returned by Catalog searches (e.g., /catalog/search).
# Restart required. Default: not set.
external.hostname=<FULLY_QUALIFIED_DOMAIN_NAME>
CHANGE: Enable Publish to Qlik Sense by Default
Publish to Qlik Sense will now be enabled by default.
# Setting this to true will start showing 'Publish to Qlik' option in Podium UI cart checkouts.
# Default: true
is.publish.to.qlik.enabled=true
DELETION: Publish to Qlik Sense: Multiple Sense Server Support in User Interface
Configuration of specific Sense servers for Publish to Qlik Sense has been moved to the Catalog user interface. General settings remain in core_env.properties.
The following is taken from the "Qlik Catalog + Qlik Sense Integration Guide". The nine, new UI fields for Publish to Qlik Sense were formerly in core_env.properties. The following properties can now be retired (either commented-off or removed from core_env.properties):
The following two properties were also removed and have been superseded by pre-existing Qlik Sense Connector fields:
The following global settings remain in the core_env.properties file -- please see the comments there:
CHANGE: Enable Catalog Auto-Creation of Sense Security Rules & Auto Add Newly Discovered QVDs to Local/AD Groups
The property controlling these capabilities is now named "qlik.sense.invert.security.model". It was formerly named "qlik.sense.auto.create.security.rules" when introduced in the initial February 2022 release. See earlier description of QDCB-1114.
# Normally, on QVD import, Catalog creates Groups that shadow Sense Data Connections. QVD entities are then added to
# these Groups. A Sense admin creates Security Rules granting Users access to Data Connections. When a User logs
# in to Catalog, a security audit is conducted against Sense, and the User is added to Groups if they have access to the
# corresponding Data Connections.
#
# This model may be inverted. A Catalog admin can instead manually add QVD entities to Catalog local/AD groups. When
# this occurs, users running Publish to Qlik Sense may need Data Connection Security Rules created in order to load
# published data in Sense. Catalog can be configured to automatically create any needed Security Rules by setting this
# property to true. In addition, this property ensures that once a single QVD entity has been manually added to a Group,
# future QVD entities discovered during import in the same Data Connection folder will be automatically added to the
# same group.
#
# Properties 'qlik.sense.root.admin.directory.name' and 'qlik.sense.root.admin.user.name' must also be set.
# Formerly: qlik.sense.auto.create.security.rules. Default: false
#qlik.sense.invert.security.model=true
ADDITION: Enable Qlik Catalog Auto-Creation of Sense Security Rules
Set this property to true to have Catalog auto-create Qlik Sense Data Connection Security Rules (if needed) as part of the Publish to Qlik Sense process. See earlier description.
# If Catalog local or AD Groups have had QVD Sources/Entities added to them, users running Publish to Qlik Sense
# may need Data Connection Security Rules created in order to load data in Sense. Normally, these Security Rules
# should be created and managed in Sense QMC. However, Catalog can be configured to automatically create any
# needed Security Rules if this property is set to true. Properties 'qlik.sense.root.admin.directory.name' and
# 'qlik.sense.root.admin.user.name' must also be set. Default: false
#qlik.sense.auto.create.security.rules=true
ADDITION: Alter SAML Identity Provider User Domain Name
Specify this property to set or alter the domain name of the user sent by the SAML identity provider (IdP) to Catalog. See earlier description.
# If the IdP does not append a domain and one is needed, or you wish to change the domain, set this property. If the
# property is set to a value of "test.com", a SAML principal name of "jdoe" would become "jdoe@test.com"; a SAML
# principal name of "jdoe@other.com" would become "jdoe@test.com". The known use case for this property is to match
# principal (aka user) names coming from the IdP with the users imported from Active Directory. Default: not set
#saml.alternate.domain=
No changes.
CHANGE: Publish to Qlik Sense Enhancement
The Publish to Qlik Sense RootAdmin user is now more widely applied. Before, it was only used when multiple domains were specified in property "qlik.sense.active.directory.name". Now, it is used to ensure a known, valid Sense user is being used for Publish to Qlik Sense. See earlier description of QDCB-1007.
# Enter the directory and user name of a Sense 'RootAdmin' user.
# Used to validate that the domain user being used for Publish to Qlik Sense has previously logged into the
# Sense server. This prevents users known only to Catalog being inadvertently created in Sense.
# Mandatory if multiple directories were specified in property 'qlik.sense.active.directory.name'.
qlik.sense.root.admin.directory.name=AD
qlik.sense.root.admin.user.name=sense-service
ADDITION: Extended Support for Fields in Prepare Dataflows that are also Pig Reserved Words
Prepare dataflow jobs will fail if fields are named using unanticipated Apache Pig reserved words. Such words can now be configured. See earlier description of QDCB-1107.
# Entity fields used in Prepare Dataflows may also be Pig reserved words (e.g., STORE). Frequently used reserved words
# are correctly handled if they are field names. This property may be used to augment the set of known reserved words
# with unanticipated words. Words must be comma separated. Default: not used
#pig.reserved.words.additional=register,CASE
Qlik Catalog November 2022 SR1 - Application
Qlik Catalog November 2022 SR1 - Installer
About Qlik
Qlik’s vision is a data-literate world, where everyone can use data and analytics to improve decision-making and solve their most challenging problems. A private SaaS company, Qlik offers an Active Intelligence platform, delivering end-to-end, real-time data integration and analytics cloud solutions to close the gaps between data, insights and action. By transforming data into Active Intelligence, businesses can drive better decisions, improve revenue and profitability, and optimize customer relationships. Qlik does business in more than 100 countries and serves over 50,000 customers around the world.