Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.
We want to highlight some of the cool things now available in the Qlik Replicate and Qlik Enterprise Manager May 2022 Client Managed Releases.
Endpoint Improvements
Salesforce Incremental Load (Source)
We have introduced support for incremental loading from all Salesforce objects which provides a greater flexibility from this source endpoint with respect to objects and fields replicated.
The Salesforce (Incremental Load) uses the salesforce Bulk API v2 which is based on change polling and as has the advantage of supporting almost all tables, compared to some limitations in the Salesforce (Streaming CDC) method in the Salesforce endpoints. You can use this method if you need to capture changes from tables that are otherwise not supported with Salesforce (Streaming CDC).
Additionally, driven by customer feedback you can also leverage “query like” incremental loads via the Salesforce BULK and SOAP APIs which has support for calculated and compound fields.
Parquet File fFormat for Amazon S3 (Target)
Again, based on customer demand, we are pleased to announce that the Amazon S3 endpoint now supports the Parquet file format, enabling a simple “file based” raw lake for data science use cases. Parquet is an open-source file format built to handle flat columnar storage. It’s well known for both its performant data compression and the ability to handle a wide variety of encoding types. The addition of Parquet will provide optimizations in Qlik Replicate that speed query execution due to its efficient file format. We hope to be able to soon support Parquet file format in other BLOB storage like ADLS and GCP. Stay Tuned!
Microsoft Azure SQL (MS-CDC) (Source)
The Microsoft Azure SQL (MS-CDC) endpoint now supports capturing changes from Azure SQL Database using Microsoft Change Tables. This effectively adds Azure SQL Database as a ‘new source’ enabling secure, real-time data replication from Azure SQL Database systems in the cloud. We are working towards certification for “bare metal” SQLServer as well, so keep an eye out for that announcement soon.
Security
SAP Extractor (Source)
The May 2022 release continues our support for various authentication methods. You can now connect to an SAP Server using Secure Network Communication (SNC). You can enable the new SNC option in the SAP Extractor endpoint's General tab.
Google Cloud BigQuery (Target)
With this target endpoint we have added tighter security options by supporting the role “Job User” which can load data without requiring editing rights.
When you are wanting to replicate to Google Cloud BigQuery you can now choose the following permissions:
- BigQuery > BigQuery Data Editor
- BigQuery > BigQuery Job User
Support for Azure Government Cloud and China Cloud
As part of our ongoing commitment for supporting our public sector customers we have introduced support for connecting to Azure resources on Microsoft’s Azure Government Cloud or China Cloud when using the following target endpoints:
- Microsoft Azure ADLS
- Microsoft Azure Event Hubs
- Microsoft Azure HDInsight
- Microsoft Azure Synapse Analytics
- Microsoft Azure Databricks
- Microsoft Azure Databricks Delta
- Snowflake on Azure
Performance
SAP Improvements
We have enhanced our SAP endpoints in the May 2022 release which will offer significant improvements in performance.
SAP Extractor Based
Particularly interesting for customers with an SAP runtime license looking to improve their latency times. Some of our ‘early adopter’ customers are seeing up to 13x improvements.
SAP Application Support via TriggersFor those SAP customers with an SAP Enterprise License, also in our early access trials, customers are reporting improved performance up to 50% faster and improved SAP source utilization, with some customers managing up to 60% cost savings.
If you are interested in working with the new SAP Extractors or SAP Triggers please reach out to your Qlik account team. While the new SAP Extractors and Triggers are fully supported, they are not yet generally available to all customers.
Google Cloud BigQuery and Snowflake INSERT Optimization
Following on from adding support for MERGE in previous releases, May 2022 release now supports an optimization for INSERT only activity which can significantly help to improve costs when moving data to a cloud target.
A The new “Optimize inserts” option is available in added to the “Change Processing Tuning” tab of “Task Ssettings”. To use the new option, ”Apply changes using SQL MERGE” must also be turned on.
With these options turned on when a batch of change records consist of INSERTs only. Qlik Replicate will perform INSERTs instead of using SQL MERGE, which will help to improve performance and reduce costs for these types of workload.
In this release the “Apply changes using SQL MERGE” and “Optimize inserts” options are currently only available for tasks configured with the following target endpoints:
- Google Cloud BigQuery
- Snowflake on Azure, Snowflake on Google, and Snowflake on AWS
Notification Type and API
We are always striving to help you to be more proactive when monitoring and managing your Qlik Data Integration environments and to that end May 2020 2022 releases for both Qlik Replicate and Qlik Enterprise Manager incorporate a new Notfication notification type and an API improvementa new API.
New task notification for CDC interruptions
The new “No changes were captured in the last <n> minutes notification” will alert you if no changes were captured from the source database within the specified time limit. With most databases, receiving this notification will usually indicate a connectivity problem. However, it is worth noting that with the following databases, it could simply mean that no changes occurred during the specified period.
- Microsoft SQL Server
- IBM DB2 for LUW
- IBM DB2 for iSeries
When the task resumes capturing changes, an additional notification that change capture has resumed will be sent, and the notification timer will be reset.
Support for replacing endpoint properties using the Qlik Enterprise Manager API
This version adds the ability to edit specific endpoint properties using the Enterprise Manager API. When not editing the value directly in the user interface, this was previously only possible using export, edit, and import operations. The functionality is available through the new PatchEndpoint method, which is available in all of the APIs (REST, .NET, and Python).
We hope you enjoy trying out the new releases and would love to hear your feedback and success stories, especially in the improvement gains.
As always if you would like to try out this release. Go to the Qlik Downloads Site and filter for “Qlik Data Integration”, and then select the product and version you would like to download.