A new release of Qlik Data Integration is here, with exciting updates to Compose. These updates address our customers’ needs across data lake creation and improved warehouse automation.
Let us start with the top improvements to data warehouse automation.
Flexibility to manage change data. When the option to keep changes in change tables is selected, Compose will not delete nor archive the landed data after reading it. Instead, Compose will track which data has been read and which hasn’t. This capability allows users to manage change data to fit their use case as they can reuse change data across Compose and ETL tasks within those projects. Customers can reduce cloud data processing costs for low latency use cases by deleting data at a scheduled maintenance time.
Qlik Hybrid Data Delivery (HDD) to ingest data into DW projects. When users choose to keep changes in change tables, they can now use Qlik Cloud Data Service – Hybrid Data Delivery to land data instead of using Replicate. This option enables flexibility to access and ingest data from different previously inaccessible cloud data sources into a data warehouse and create new data marts.
Reference dimensions. Users would like to use the same dimension in several data marts by reference to manage it once and use it several times with minimal overhead. By reusing dimension tables across data marts, users can break up fact tables into smaller units for both design and data loading while ensuring data consistency for analytics. This reuse improves developer efficiency, data mart design, execution flexibility, and reuse of data sets within Compose by providing conformed reference dimensions.
Data Mart Adjustment Enhancements. As we know, data warehouse automation improves designer agility. Now designers can adjust the data mart without dropping and recreating it, so a lengthy, costly reload of the data mart can be saved. Now users can flexibly control how to handle data mart changes to increase data availability and improve project deployment capabilities.
Second, let us examine improvements to simplify data lake creation.
Faster Execution with Databricks Photon. Databricks 9.1 LTS is now certified for Qlik Compose 2021.08 and above. We recommend enabling Photon in your Databricks clusters to boost your existing workloads to execute faster and reduce your data lake query costs.
Live View Performance Enhancements. We now alter Live Views with each change data capture run to ensure pruning Change Table partitions for optimal performance.
Try out the release by going to Support > Downloads on Qlik.com and filtering your options to Qlik Compose version 2022.5.
Check out this eBook for your options for optimizing your experience with data lakes and data warehouses.
Register for the upcoming Data Integration Roadmap Session on September 16 at 11 am EST. Register here