Qlik Data Integration November 2022 Client-Managed Releases - now available

[QlikProductUpdates]

November is known for celebrations and fireworks in many countries across the world. Perhaps that is why November is also the month that brings a host of releases across the Qlik Data Integration Client-Managed portfolio. This is a roundup of just a few of the highlights.

[QlikProductUpdates]

Qlik Replicate November 2022 Client-Managed

The November 2022 release is packed with many exciting announcements. We have new endpoints, capabilities, improvements, and optimizations.

New Endpoints and EnhancementsThe Qlik Replicate November 2022 release brings several new endpoints and enhancements, some were made available as part of an earlier Service Release (May2022 SR) and are also included in this General Availability release.

SAP New Endpoint & Optimizations  supporting many ways to enable data replication from SAP with Extractors and best-in-class log-based and trigger-based CDC, in this November release we have continued to expand on the many ways you can connect with your SAP data.   You will find:

• New SAP ODP endpoint as a Source - Introducing support for the SAP Operational Data Provisioning (ODP) framework that can provide a better experience than SAP Extractors and operate under an SAP Runtime license.
• SAP HANA multi-tenants
• SNC authentication

Improved Databricks Endpoints as a Target   Databricks is an important strategic technical and commercial partnership for Qlik, and we continue to improve our integration with the Databricks ecosystem and the user experience by consolidating the Databricks Lakehouse (Delta) endpoints as a target, supporting Full Load, Store Changes and Apply Changes. Both endpoints now support Databricks SQL compute as a lower cost compute layer in Databricks.

• Databricks Lakehouse (Delta) as a Target -   This release adds support for both managed and unmanaged tables in this endpoint. We have also added support for Databricks Lakehouse (Delta) on AWS and Google Cloud Platform (GCP). For a better user experience, we have consolidated the three Databricks endpoints (Databricks on AWS, on GCP and Azure Databricks) into this single endpoint.  Note: When upgrading to this release, any existing tasks using Azure Databricks Delta endpoints will be automatically migrated to the new Databricks Lakehouse (Delta) endpoint.
• Databricks Cloud storage as a target –  This endpoint is typically used with Qlik Compose to manage the data lake and historical data sets.   With this release, this endpoint and Qlik Compose now support Databricks SQL Compute, along with parquet file delivery for optimized performance with Qlik Compose.

MongoDB on ATLAS as a Source  The Qlik Replicate November 2022 release supports cloud-based MongoDB Atlas, as an additional option to the existing MongoDB endpoint.

MS SQL Server (CDC) as a Source  This release sees support for Microsoft’s MS CDC API which has recently come out of beta. This is a new option for the existing endpoint which adds significant value by simplifying complexities for example;

• Reading the lobs directly from the CT table without the need to do lookup.
• In addition to supporting sysadmin credentials, you can now choose to work with db_owner

New :

• AWS PrivateLink added - Improved security for Amazon S3 with the support for AWS PrivateLink which will help improve secure connections for larger AWS environments.  Stay tuned for support of other targets in later releases.
• Parquet file format support added for Azure Data Lake Storage Gen 2 (ADLS Gen 2) and Google cloud
  The Parquet file format provides optimizations that speed query execution due to its efficient file format. Parquet is an open-source file format built to handle flat columnar storage that enables a simple “file based” raw lake for data science use cases. It’s well known for both its performant data compression and the ability to handle a wide variety of encoding types. Aligning with our multi-cloud strategy, support for Parquet now spans across the clouds, building on Parquet support for S3 added in the Qlik Replicate May 2022 release Qlik Replicate now supports the three main Hyperscalers (AWS, Azure and GCP).

• Google BigQuery enhancements -
  This endpoint now offers added Proxy support, Use-compression as an option and the ability to allow you to set any Google Region when selecting Google Locations. In most cases compression can usually improve performance, however in some cases performance could be degrade, when trying to compress an already compressed file, for example. Therefore, we have added an option to enable you to decide if you want Qlik Replicate to use compression or not when moving data to Google BigQuery.  Google are regularly adding new Data Location Regions and if your chosen location is not available in Qlik Replicate’s built-in Location list, you can now select “Other”, and manually add the new Region code.

• Added Support for Primary Key (PK) changes
  As any DBA knows making changes to the PK directly is a big “No, No!”. To ease downstream processing, and support best practices, this Qlik Replicate Engine improvement benefits Data Lake and streaming targets, e.g., Kafka, by turning updates to the PK into a delete of the old key and an insert of the new key.

Qlik Replicate is now using FIPS compliant libraries  Qlik Replicate November 2022 release now includes the option to use FIPS compliant libraries over standard OpenSSL. FIPS (Federal Information Processing Standards) is a set of standards developed by the United States Federal Government for use in computer systems. By selecting this option in Qlik Replicate, fully compliant FIPS 140-2 SSL libraries will be used that meet a set of stringent security requirements that are associated with these libraries.

Qlik Compose May 2022 - Service Release (22.05)

Qlik Compose’s latest release offers exciting new capabilities, such as:

• Streamlined Audits - Qlik Compose maintains the history of all transactions in the Historical Data Store (HDS). When customers audit individual transactions that occur at sub-second frequency, they take the extra step of reviewing those changes in the landing area of a Data Warehouse instead of directly in the HDS. With the latest release, we are enhancing our data governance capabilities by adding support to view and manage all changes in the HDS resulting in streamlined audits.
• Expanded Automation – Currently, Qlik Compose administrators manage users and groups manually using the user interface. This Qlik Compose release augments administration capabilities via the command line interface. Administrators can further automate their Continuous Integration & Development (CI / CD) pipelines by creating and managing access to Qlik Compose using command line scripts.
• Lower Lakehouse Costs – Qlik Compose customers can now execute their automated workloads on Databricks SQL Warehouse and benefit from lower-cost computing on the latest 10.4 LTS Databricks Lakehouse Platform. Also, Qlik Compose can now consume parquet files landed by the November 2022 release of Qlik Replicate.
• Greater Agility – Qlik Compose generates SQL code to automate the lifecycle of data warehouses and different types of data marts. Additional run-time clauses are added to the generated SQL code in the latest release to reduce the number of statements executed when there is no data to process. This optimization of SQL statements for Data Warehouse projects greatly enhances data warehouse agility when minimal data changes occur.
• Windows Modernization - Compose is now certified to run on Windows 2022. Data Warehouse and Data Lake automation efforts can now utilize Qlik Replicate and Compose on this latest Windows version.

 Qlik Catalog November 2022 Client-Managed

• Schedule data loads  Qlik Catalog November 2022 removes the requirement that external tools are needed to load data on a predefined schedule. Qlik Catalog now provides the ability to define, schedule, and automatically run data loads. 
• Publishing business names  Previously, business names that were added to a data field could only be included when publishing to a Qlik target.  We’ve now expanded this capability to allow business names to be included when publishing to any target.
• Increased performance  We’ve improved the performance of incremental loads by only pulling changed data into Qlik Catalog. 
• Updating metadata tags  When importing metadata tags, one could previously only create new tags in Qlik Catalog. Qlik Catalog November 2022 gives you the option to alternatively update existing metadata tags during import.

We hope you enjoy using these new releases and would love to hear your feedback and success stories, especially in any improvement gains you achieve.

To learn about everything that is included in these releases be sure to check out the Release notes which are available here

To obtain any of these releases, go to the Qlik Downloads Site in community and filter for “Qlik Data Integration”, and then select the product and the versions you would like to download.

Note: For most products, please select the version “November 2022” or in the case of Qlik Compose please select the latest Service Release (SR) version.

 For additional help in using the downloads site - watch this handy video guide here:
How-to-download-Qlik-Products