Release Notes

[djenkins-qlik]

Table of Contents

• What's new in Qlik Catalog February 2021 Service Release 2
• Resolved Defects
• February 2021 Service Release 2
• Publish to Qlik Sense of QVD Entities Containing Mixed-Case Fields
• February 2021 Service Release 1
• Hide Source and/or Security Module for Roles including Prepare Access
• Use business name as primary entity/ field name
• February 2021 (Initial Release)
• Source, Entity and Field Names May Now Contain Foreign, Multi-Byte and Punctuation Characters (singl...
• Deferred Deletion of Publish-to-Qlik Data Loaded On-Demand
• Label change on QVD Import screen
• Known issues and limitations
• Postgres table definition and entity definition differs upon updating field types
• Upgrade notes
• New Upgrade Processes
• New properties and behavior changes
• NEW
• NEW ARGUMENT (see qvd.conversion.temp.directory)
• BEHAVIOR CHANGED
• RENAMED
• DESCRIPTION IMPROVED
• Deferred Deletion of Publish-to-Qlik Data Loaded On-Demand
• Core Env and qdc_qvd_2_csv.sh changes
• License Agreement option in the support dropdown is now configurable
• Change to Catalog Search
• New Property Allowing Compression to be Disabled During Prepare
• Downloads
• Qlik Catalog February 2021 SR2 - Application
• Qlik Catalog February 2021 SR2 - Installer

The following release notes cover the versions of Qlik Catalog released in February 2021. 

What's new in Qlik Catalog February 2021 Service Release 2

Resolved Defects

February 2021 Service Release 2

• QDCB-947 – Publish to Qlik Sense of QVD Entities Containing Mixed-Case Fields

• QDCB-948 – Publish to Qlik Initiated by User with Hyphen in Name

Publish to Qlik Sense of QVD Entities Containing Mixed-Case Fields

Jira ID: QDCB-947
When Publish to Qlik was invoked for a QVD entity, an issue arose if the QVD fields contained any upper-case characters. Previously, the script passed to Qlik Sense used "LOAD *". However, a recent change explicitly lists each field, for example, "LOAD field1, field2". The fields included in the LOAD statement were all lower-cased, causing the LOAD statement to fail in Qlik Sense if the fields contained any upper-case characters.

The fix is to preserve the original case found in the QVD file when constructing the LOAD statement to pass to Qlik Sense.

Publish to Qlik Initiated by User with Hyphen in Name

Jira ID: QDCB-948
If the logged-in Catalog username contained a hyphen "-" or other non-alpha-numeric character besides underscore "_", Publish to Qlik Sense failed. It failed because the Publish dataset and job names embed the username -- the hyphen character is not allowed in that context.

The fix was to transform the username, replacing any problematic characters with underscore, prior to embedding the username in the Publish dataset and job names.

February 2021 Service Release 1

• QDCB-761 – Hide Source and/or Security Module for Roles including Prepare Access

• QDCB-919 - Use business name as primary entity/ field name

Hide Source and/or Security Module for Roles including Prepare Access

Jira ID: QDCB-761
In order to run Prepare workflows, the access masks created for this custom role require access to the source and security modules as prepare adds data (targets) and permissions (group security on targets). Some customers have requested an ability to restrict access to those modules.

Two new core_env props have been added to forcefully hide security and source section for specific role names.

The core env props are:

# The following property works on top of QDC's security mechanism. If a user has access to the source
# section because of the roles they is associated with and there is a need to hide the source section, then
# the following property can be used. QDC will loop over all the roles logged in user is part of, and if
# any of the below roles(separated by comma) are found in the logged in user's roles, then the source module will not be displayed to them.
# Value for this can be a CSV. Default:
#roles.to.exclude.source=

# The following property works on top of QDC's security mechanism. If a user has access to the security
# section because of the roles they is associated with and there is a need to hide the security section, then

# the following property can be used. QDC will loop over all the roles logged in user is part of, and if

# any of the below roles(separated by comma) are found in the logged in user's roles, then the security module will not be displayed to them.

# Value for this can be a CSV. Default:

#roles.to.exclude.security=

As an example, if there is a custom role named 'PREPARE_EXECUTOR' with an access mask of 251721703, which gives it access to the source, prepare, and security modules. However, you want to restrict it from accessing the source module but allow it to access the security module.

roles.to.exclude.source=PREPARE_EXECUTOR

roles.to.exclude.security=

Use business name as primary entity/ field name

Jira ID: QDCB-919

Historically, technical name was used as the primary entity/ field name. With the addition of the following property, users can now use business name as primary entity name on catalog, and business name as primary field name while viewing the sample data. Business name column has also been added to the 'Field Statistics and Metadata' table in 'Catalog Details' view.

Property to configure if you want to display business or technical name at catalog, sample data The value can be BUSINESS or TECHNICAL . Default: TECHNICAL #display.entity.name.preference=TECHNICAL

February 2021 (Initial Release)

• QDCB-573 – Source, Entity and Field Names May Now Contain Foreign, Multi-Byte and Punctuation Characters (single-node only)
• QDCB-590 - Deferred Deletion of Publish-to-Qlik Data Loaded On-Demand
• QDCB-696 - Label change on QVD Import screen.
• QDCB-712 - Core Env and qdc_qvd_2_csv.sh changes
• QDCB-740 – P2Q Select statement not explicit
• QDCB-756 - License agreement in the Support dropdown
• QDBC-773 – Prepare Dataflow Editor not Saving Added/Deleted Fields
• QDCB-826 – License Agreement option in the support dropdown is now configurable
• QDCB-887 - QVD Import fails due to long names in Postgres (63+)
• QDCB-901 – Change to Catalog Search
• QDCB-915 – New Property Allowing Compression to be Disabled During Prepare

Source, Entity and Field Names May Now Contain Foreign, Multi-Byte and Punctuation Characters (single-node only)

Jira ID: QDCB-573
Source, Entity and Field names may now contain foreign, multi-byte and punctuation characters. This enhancement is for the single-node version of Catalog. This enables JDBC, QVD and other loads of columns containing characters such as '#' (e.g., "F2#1"), and schema and table names containing foreign characters.

The external Source, Entity and Field names in the "Source" module now exactly match their schema, table and column JDBC counterparts. For example, Cyrillic and Chinese characters may be used in imported table names -- they now properly appear in the Catalog UI.

The matching internal Source, Entity and Field names in the "Discover" module support virtually the same character set with the exception of the following characters: ./":'{}[],\. These eleven characters will be converted to "_" (underscore) in the Discover module -- all other characters will be preserved and appear in the UI. The conversion to underscore allows these names to be used in file-system directory names (where data is "landed") and in a variety of scripting and database usages. Queries against the Catalog distribution database (e.g., from the Discover / Query tab) should double quote schema and table names containing non-alphanumeric characters.

Support for these characters extends across the entire Catalog product, including the Prepare and Publish modules. The only special handling required is when referencing Field names in Prepare Pig expressions: Field names with non-alphanumeric characters must be double quoted. For example, if a Field (column) was named "F2#1", to reference it in the Pig CONCAT expression, double quotes must be used: CONCAT("F2#1", '-suffix').

Deferred Deletion of Publish-to-Qlik Data Loaded On-Demand

Jira ID: QDCB-590
When core_env property "podium.qlik.dataconnection.addressed.registered.entity.generating.from=INTERNAL", REGISTERED and ADDRESSED JDBC entities are loaded on-demand prior to Publish-to-Qlik Sense. At the completion, the entity is "demoted" and its files and distribution tables are scheduled for deletion. The asynchronous garbage cleaner runs on a cycle of five minutes, meaning the data may be deleted before the Qlik Sense load is completed.

The deletion of these records is now deferred for eight hours. This can be adjusted in core_env.properties:

# For entities that are ad-hoc promoted as part of Publish to Qlik, the follow-on deletion (aka demotion) of files and 

# distribution database tables will be deferred for this many hours. This prevents the Garbage Cleaner from 

# inadvertently deleting data while it is being loaded into Qlik Sense. Fractional hours may be entered. Default: 8 hours. 

#qlik.sense.data.deletion.deferral.period.hours=8

Label change on QVD Import screen

Jira ID: QDCB-696

While doing the import of QVD entities along with their data, the status column in the "View Details" modal of a single QVD import run only tells about the status of the scan and not the data load.

The label from "Status" to "Scan Status".

Core Env and qdc_qvd_2_csv.sh changes

Jira: QDCB-712
qvd.openconnector.script.path property of core_env has been modified to include a temp directory parameter..

qvd.openconnector.script.path=qlikcore-integration/qdc_qvd_2_csv.sh %prop.qvd.file.linux.folder %prop.qvd.file.entity.original.name %loadingDockLocation %loadingDockUri %qlikCoreScript %tempDir

A new core_env property has been introduced as well: qvd.conversion.temp.directory

P2Q select statement not explicit

Jira: QDCB-740
The Publish to Qlik (P2Q) script for Managed entities and QVD entities now supports aliases. If you want to provide alias to your column names in the script, please modify the fields to add business name into them - the business name is then used as an alias.

If the field doesn't have business name, it will not have an alias.

Also, please note that this change doesn't work with non-managed entities as the script for them is picked up from the src.file.glob property and is not created via the code. 

Also, please note that the LOAD script for QVD entities now defines the columns explicitly rather an asterisk.

License agreement in the Support dropdown

Jira ID:  QDCB-756

We have now added the Qlik User License Agreement in the 'Support' button in the UI.

Prepare Dataflow Editor not Saving Added/Deleted Fields

Jira ID: QDCB-773
The Prepare Dataflow "Save" action was not persisting target Entity field additions and deletions. If fields were added or deleted from a target Entity, the "Save" action failed to include the modified Entity with the HTTP API request. This was due to a Catalog user interface (UI) JavaScript bug. This issue has been fixed, and it is now possible to add and delete fields on a Prepare Dataflow target Entity.

License Agreement option in the support dropdown is now configurable

Jira ID: QDCB-826
The change can be controlled via two new  core_env props:

# The value of this property is used as a label in the drop/down. Even though there are no invalid/ names

# We think the value of this property will mostly say License Agreement OR Terms of Use

# Default: License Agreement

#license.agreement.menu.display=License Agreement

# This is the document that comes up when the button with the above text is clicked from the UI.

# This document must be put in the <apache_tomcat>/<web_apps>/<qdc_web_app>/resources folder.

# The path will then be set as resources/<document_name> as below

# Default: resources/QULA.pdf

#license.agreement.text.display=resources/QULA.pdf

QVD Import fails due to long names in Postgres (63+)

Jira ID: QDCB-887
QVDs with entity names containing more than 63 characters were failing import due to a missing customization within the Postgres installer that is packaged with Catalog. The Postgres installer has been modified to support QVD import of entity names which contain up to 256 characters.

Change to Catalog Search

Jira ID: QDCB-901
Previously, the search query used to build the data displayed on the Catalog home page used multiple SQL "like" operators. The search term was concatenated with both a leading and trailing wildcard "%" character, denoting "contains". This resulted in an expensive SQL query.

Now, the default behavior is to only add a trailing "%" character, denoting "starts with". This is quicker.

If the customer desires the original behavior, they can either manually prepend a "%" character to the beginning of the search term or set core_env property "prepend.leading.percent.sign.for.catalog.search" to true.

The new property:

# Flag to control if the searched term in Catalog will have a percent sign prepended to it implicitly.

# The term always has a trailing percent sign appended to it implicitly. If the value of this flag

# is set to false, the user must put the percent sign (%) themselves in the search term for the code

# to do wildcard search. Default: false

#prepend.leading.percent.sign.for.catalog.search=false

New Property Allowing Compression to be Disabled During Prepare

Jira ID: QDCB-915
On at least one multi-node Catalog install, Prepare dataflows fail when run in LOCAL mode due to a native compression library issue unrelated to Catalog (AWS EMR installs, snappy library).

Set core_env property "prepare.mapreduce.map.output.compress" to "false" to disable the use of compression during Prepare dataflow LOCAL execution.

The property:

# When running Prepare in LOCAL mode, this property controls whether Pig intermediate Map job output is compressed.

# On some multi-node environments, there have been problems with native compression libraries. Setting this to false

# avoids the use of compression. True means the Pig default will be used for the environment. False explicitly disables

# compression. Default: true

#prepare.mapreduce.map.output.compress=true

Known issues and limitations

Postgres table definition and entity definition differs upon updating field types

 Jira ID: QDCB-8265
For a loaded entity, if the field types are changed (for example modifying external metadata from float to int), and it is reloaded, the Postgres table fields will show incorrectly, still using the old data types for definition of the Postgres foreign table, though the two should resync.

Upgrade notes

New Upgrade Processes

Do not attempt to upgrade until the following is understood.

If upgrading from a version of Qlik Catalog prior to September 2020 there are utilities that MUST be run after Catalog is upgraded to February 2021 (4.9 release). The server may not start until the first two have been run and will log a WARN at startup until the third is run. Do NOT upgrade the server until familiar with these utilities and the information required to run them. It will take time to gather this information.  Gathering the information BEFORE Catalog is upgraded will minimize downtime.

Run the utilities in this order:

1. jwt2CertsUtility -- please review readme.txt

This will be required if Qlik Sense Connectors have been defined in order to load QVDs.

Will need to gather networking info and certificate files from Qlik Sense servers.

May be run from any directory.

1. singleNodeUpgradeForEntitiesWithBadOrUglyData.sh -- please review comment in script

This will be required if the installation is single-node.

Will need podium_dist database info if defaults altered.

May be run from any directory.

1. singleNodeUpgradeToGrantReadOnlyUserAccessToDistSchemas.sh -- please review comment in script

This will be required if the installation is single-node.

Will need podium_dist database info if defaults altered.

May be run from any directory.

New properties and behavior changes

NEW

# This is the temporary location which houses the CSV & QVD created as a result of execution of qdc_qvd_2_csv and qdc_csv_2_qvd.sh respectively.

# The default location is set to /tmp but can be overridden using this property

#qvd.conversion.temp.directory=/tmp

# When running Prepare in LOCAL mode, this property controls whether Pig intermediate Map job output is compressed.

# On some multi-node environments, there have been problems with native compression libraries. Setting this to false

# avoids the use of compression. True means the Pig default will be used for the environment. False explicitly disables

# compression. Default: true

#prepare.mapreduce.map.output.compress=true

# For entities that are ad-hoc promoted as part of Publish to Qlik, the follow-on deletion (aka demotion) of files and

# distribution database tables will be deferred for this many hours. This prevents the Garbage Cleaner from

# inadvertently deleting data while it is being loaded into Qlik Sense. Fractional hours may be entered. Default: 8 hours.

#qlik.sense.data.deletion.deferral.period.hours=8

# The value of this property is used as a label in the drop/down. Even though there are no invalid/ names

# We think the value of this property will mostly say License Agreement OR Terms of Use

# Default: License Agreement

#license.agreement.menu.display=License Agreement

# This is the document that comes up when the button with the above text is clicked from the UI.

# This document must be put in the <apache_tomcat>/<web_apps>/<qdc_web_app>/resources folder.

# The path will then be set as resources/<document_name> as below

# Default: resources/QULA.pdf

#license.agreement.text.display=resources/QULA.pdf

# Flag to control if the searched term in Catalog will have a percent sign prepended to it implicitly.

# The term always has a trailing percent sign appended to it implicitly. If the value of this flag

# is set to false, the user must put the percent sign (%) themselves in the search term for the code

# to do wildcard search. Default: false

#prepend.leading.percent.sign.for.catalog.search=false

NEW ARGUMENT (see qvd.conversion.temp.directory)

# The Qlik Core script path to convert QVD to CSV.

#qvd.openconnector.script.path=qlikcore-integration/qdc_qvd_2_csv.sh %prop.qvd.file.linux.folder %prop.qvd.file.entity.original.name %loadingDockLocation %loadingDockUri %qlikCoreScript %tempDir

# The Qlik Core script path to convert CSV to QVD.

#qvd.publish.script.path=qlikcore-integration/qdc_csv_2_qvd.sh %sourceDir %targetDir %targetFileName %tempDir

BEHAVIOR CHANGED

# The maximum size of the connection pool used to load entities from a specific JDBC source.

# Each unique JDBC URL has its own connection pool. 0 indicates do not use a connection pool and

# instead create a new connection for each load. Default: 32

#jdbc.source.pool.size=32

# The fetch size used when loading JDBC sources. Fetch size is analogous to page size, incrementally processing

# the data. If specified, the fetch size is only altered if the JDBC driver would normally use 0 or greater than

# 1,000. Some JDBC drivers, like PostgreSQL, will read _ALL_ records into memory if left to their default of 0 --

# this can result in out-of-memory. 0 indicates do not alter the fetch size. Default: 250

#jdbc.load.fetch.size=250

RENAMED

WAS saml.logout.url

# Logout page. The user will be redirected to this page once the session expires via a timeout. You can set this to any page

# including logged-out to get a QlikSense like logout page. Please note that for SAML & DISABLED authentication modes, this property 

# is used to not only configure the landing page for session timeout, but also for logout. If the value for this is not a Qlik Catalog

# page, then the URL must start with http(s); if it's a Qlik Catalog page, it must be a jsp page name without the .jsp extension

# extension. Default: logged-out

#logout.url=logged-out

DESCRIPTION IMPROVED

# Entity type to determine if the Publish to Qlik load script query should be generating from the internal or external

# object for the managed entity. Please note that this pertains only to JDBC entities. QVDs are already sourced locally

# in Sense. Other entity types (e.g., files) can only be published from INTERNAL to Catalog.

# Choices: INTERNAL, EXTERNAL. Default: INTERNAL

#podium.qlik.dataconnection.managed.entity.generating.from=INTERNAL

# Entity type to determine if the Publish to Qlik load script query should be generating from the internal or external

# object for the addressed or registered entity. Please note that this pertains only to JDBC entities. QVDs are already

# sourced locally in Sense. Other entity types (e.g., files) can only be published from INTERNAL to Catalog. As such,

# these entities will go through adhoc promotion to managed and demotion after publish.

# Choices: INTERNAL, EXTERNAL. Default: EXTERNAL

#podium.qlik.dataconnection.addressed.registered.entity.generating.from=EXTERNAL

Deferred Deletion of Publish-to-Qlik Data Loaded On-Demand

Jira ID: QDCB-590
When core_env property "podium.qlik.dataconnection.addressed.registered.entity.generating.from=INTERNAL", REGISTERED and ADDRESSED JDBC entities are loaded on-demand prior to Publish-to-Qlik Sense. At the completion, the entity is "demoted" and its files and distribution tables are scheduled for deletion. The asynchronous garbage cleaner runs on a cycle of five minutes, meaning the data may be deleted before the Qlik Sense load is completed.

The deletion of these records is now deferred for eight hours. This can be adjusted in core_env.properties:

# For entities that are ad-hoc promoted as part of Publish to Qlik, the follow-on deletion (aka demotion) of files and 

# distribution database tables will be deferred for this many hours. This prevents the Garbage Cleaner from 

# inadvertently deleting data while it is being loaded into Qlik Sense. Fractional hours may be entered. Default: 8 hours. 

#qlik.sense.data.deletion.deferral.period.hours=8

Core Env and qdc_qvd_2_csv.sh changes

Jira: QDCB-712
qvd.openconnector.script.path property of core_env has been modified to include a temp directory parameter..

qvd.openconnector.script.path=qlikcore-integration/qdc_qvd_2_csv.sh %prop.qvd.file.linux.folder %prop.qvd.file.entity.original.name %loadingDockLocation %loadingDockUri %qlikCoreScript %tempDir

A new core_env property has been introduced as well: qvd.conversion.temp.directory

License Agreement option in the support dropdown is now configurable

Jira ID: QDCB-826
The change can be controlled via two new  core_env props:

# The value of this property is used as a label in the drop/down. Even though there are no invalid/ names

# We think the value of this property will mostly say License Agreement OR Terms of Use

# Default: License Agreement

#license.agreement.menu.display=License Agreement

# This is the document that comes up when the button with the above text is clicked from the UI.

# This document must be put in the <apache_tomcat>/<web_apps>/<qdc_web_app>/resources folder.

# The path will then be set as resources/<document_name> as below

# Default: resources/QULA.pdf

#license.agreement.text.display=resources/QULA.pdf

Change to Catalog Search

Jira ID: QDCB-901
Previously, the search query used to build the data displayed on the Catalog home page used multiple SQL "like" operators. The search term was concatenated with both a leading and trailing wildcard "%" character, denoting "contains". This resulted in an expensive SQL query.

Now, the default behavior is to only add a trailing "%" character, denoting "starts with". This is quicker.

If the customer desires the original behavior, they can either manually prepend a "%" character to the beginning of the search term or set core_env property "prepend.leading.percent.sign.for.catalog.search" to true.

The new property:

# Flag to control if the searched term in Catalog will have a percent sign prepended to it implicitly.

# The term always has a trailing percent sign appended to it implicitly. If the value of this flag

# is set to false, the user must put the percent sign (%) themselves in the search term for the code

# to do wildcard search. Default: false

#prepend.leading.percent.sign.for.catalog.search=false

New Property Allowing Compression to be Disabled During Prepare

Jira ID: QDCB-815
On at least one multi-node Catalog install, Prepare dataflows fail when run in LOCAL mode due to a native compression library issue unrelated to Catalog (AWS EMR installs, snappy library).

Set core_env property "prepare.mapreduce.map.output.compress" to "false" to disable the use of compression during Prepare dataflow LOCAL execution.

The property:

# When running Prepare in LOCAL mode, this property controls whether Pig intermediate Map job output is compressed.

# On some multi-node environments, there have been problems with native compression libraries. Setting this to false

# avoids the use of compression. True means the Pig default will be used for the environment. False explicitly disables

# compression. Default: true

#prepare.mapreduce.map.output.compress=true

Downloads

 

Qlik Catalog February 2021 SR2 - Application

Qlik Catalog February 2021 SR2 - Installer

About Qlik

Qlik’s vision is a data-literate world, where everyone can use data and analytics to improve decision-making and solve their most challenging problems. A private SaaS company, Qlik offers an Active Intelligence platform, delivering end-to-end, real-time data integration and analytics cloud solutions to close the gaps between data, insights and action. By transforming data into Active Intelligence, businesses can drive better decisions, improve revenue and profitability, and optimize customer relationships. Qlik does business in more than 100 countries and serves over 50,000 customers around the world.

qlik.com