Introduction

In 2023 Qlik acquired Talend to further expand our data integration portfolio and Qlik is in the process of integrating Talend technology  into our Data Integration stack. This document is not looking at the future direction of our products and only focuses on solutions that can be delivered in Qlik Cloud Data Integration at the time of writing (May 2024). Qlik is constantly adding new capabilities to Qlik Cloud and we encourage you to speak to your Qlik representative if you are looking at a use-case not covered here.

Scenario 1:  Data Pipelines for analytics, AI and Machine Learning

As a market leader in the analytics space, Qlik provides analytics solutions to tens of thousands of organizations. But before customers can implement analytics solutions, they first need to make that data available to their analytics platform. Whether a customer uses Qlik Analytics, another vendor’s solution or a combination, Qlik Cloud Data Integration can extract the data, transform it into an analytics ready format and make the data available to the desired platform. 

Qlik Cloud Data Integration supports multiple architectural patterns in support of data pipelines.  These are enabled by on 2 core functional areas:

Data Movement

• Ingest data from relational databases, SAP and mainframe systems leveraging log-based changed-data-capture. This provides a low-latency, automated method of ingestion with very little impact on the source system.
• SaaS applications are also supported for automated ingestion and data movement. Qlik Cloud Data Integration provides packaged connectors for SaaS applications which remove complexity of integrating with REST APIs by automatically discovering the sources metadata and delivering a normalized data structure in your target

Data Transformation

• ELT (pushdown) transformations support any transformation workload on data ingested by Qlik Cloud Data Integration or another tool of your choice.
• Transformation flows provide a graphical interface for designing transformations which is complemented by a code-first SQL approach that allows for extensibility to support any use case.
• Automated data marts provide a no-code method to automate fact and dimension creation.   This reduces the time to create a data mart by automating denormalization, late arriving dimensions, type 2 processing and many of the requirements of a data mart.

Some specific examples of this are provided below.

Data Warehouse Automation

Data warehouse automation is the process of transforming data from source systems into an analytics ready format. Data warehouse automation with Qlik Cloud Data Integration uses a no-code/low-code approach to populate the data warehouse and/or data-marts including the creation of Fact and Dimension tables, handling slowly changing dimensions, de-normalization and other artifacts as required. Integrated data movement, transformation and data mart automation capabilities ensure an end-to-end data pipeline to support data warehouse requirements. Data warehouse can be deployed in the cloud or on-premises.

Data Lake Landing

A data lake is a data storage strategy whereby a centralized repository holds all of your organization's structured and unstructured data. It employs a flat architecture which allows you to store raw data at any scale without the need to structure it first.

Qlik Cloud Data Integration provides a zero-code approach for quickly landing your data in SaaS based data lakes. Targets supported today are object based storage from the major cloud vendors.

Data Lakehouse Automation

A Data Lakehouse is a hybrid between a data warehouse and a data lake, providing many of the benefits of both without their limitations. This is achieved by providing a metadata layer, SQL and other features of a traditional data warehouse as well as the ability to handle semi structured and unstructured data like a data lake. Data Lakehouses often subscribe to a medallion or three-tiered architecture.  Qlik Cloud Data Integrations core features support defining and managing these layers.   

• Data movement automatically handles a type 2, source-oriented ‘bronze’ layer.  
• Integrated, fit-for-purpose transformation capabilities support creation and management of the silver layer.
• The combination of fit-for-purpose transformations and data mart automation enable the creation of the gold or consumption layer of the data lakehouse.   

Qlik Cloud Data Integration fully supports all the major cloud data lakehouse platforms.

Enabling Qlik Analytics

If Qlik Cloud Analytics is your chosen analytics platform, Qlik Cloud Data Integration can manage the creation and updating of QVDs used in Qlik Cloud Analytics. We refer to these as Active QVDs as they will have changes applied automatically by Qlik Cloud Data Integration, without any coding on either the Data Integration or Analytics side. QVDs will be stored either within a Qlik Cloud Data Space, or in your own S3 storage. If using your own storage, the QVDs can also be consumed by client-managed deployments of Qlik Sense and QlikView.

Enabling machine learning and AI

In the previous section we discussed QVDs in the context of Analytics, but QVDs can also be consumed by AutoML as part of a machine learning experiments. AutoML can be scheduled to run at certain times, but to only run if the dataset has changed. This ensures that the predictions you get from AutoML are always based on the latest data.

Scenario 2: Data Replication for Operational reporting and query offload

Many organizations are challenged with resource constraints and at times performance concerns with business-critical operational systems. Read only applications, queries and operational reporting solution requirements often have a detrimental impact to these systems and can (for example in the case of mainframe) lead to increased costs. While many replication solutions require the destination database to be kept offline, that is not the case with Qlik Cloud Data Integration. 

Data can be replicated to the target databases in real-time which can then be used for operation reporting. The Target databases could be running on a separate database platform better suited to reporting than the original operational system. Data replication also provides for high availability of your data and can be a component of a wider disaster recovery solution.

Scenario 3: SaaS and On-premises Application ingestion and Analytics

Most organizations today use commercial software packages and/or SaaS solutions for their ERP, CRM, HR and other business functions. These products are often not well suited to analytics, reporting or other needs and often do not store their data in a format that is easy to consume to other uses. In many cases directly accessing the data is restricted either physically or by license.  Access must be via approved methods and is often technically challenging to use. Qlik Cloud Data Integration has support for more than 50 applications including Salesforce, Workday and SAP. It can make this data available for whatever you require, whether that is Analytics, integration with other systems or for migration to a new system.

Scenario 4: Cloud Migration

Many organizations are moving workloads to the cloud to leverage greater efficiency, improved scalability, and reduced deployment timeframes. However, it is crucial that during migration customers have confidence that their data is migrated accurately and securely. It may be required to run migration processes several times and in multiple environments. Minimizing the time the migration takes is crucial to minimize business disruption and downtime. To achieve these goals, it is important to automate your cloud data migration with a solution such as Qlik Cloud Data Integration.

Qlik Cloud Data Integration automates the process and can ensure that fast, repeatable, auditable processes are used to migrate data.  Depending on the customers migration requirements, they can choose to use Change Data Capture (CDC) or full reloads as appropriate. 

More Information

For more Information on Qlik Cloud Data integration see our evaluation guide Qlik Cloud Data Integration, or for in-depth technical information, see our help at Qlik Cloud - Integrating Data.

Environment  

Qlik Cloud