Qlik Data Integration Product Use Cases

Pedro_Lopez · Jan 3, 2023 5:22:22 AM

This article aims to give an overview of possible Qlik Data Integration real-life use cases.

Please note that the examples in this document are not the only applicable use cases of our Qlik Data Integration products.

Qlik Replicate

Qlik Replicate is a data integration software that empowers organizations to accelerate data replication, ingestion and streaming across a wide variety of heterogeneous databases, data warehouses, and Big Data platforms. After loading the selected tables to the target, Qlik Replicate’s high-performance change data capture (CDC) technology remotely scans transaction logs and rapidly delivers real-time data updates.

In what scenarios can it be used?

Scenario 1: Migrating Legacy Databases

Data sits in a DB2 Mainframe and the intention is to downstream it to applications that will be later consumed by end users. Considering the difficulties of developing an application that reads data from a Mainframe, Qlik Replicate can be an easily implemented solution that will transfer this data to a more traditional database such as an SQL Server or Oracle, which can then later be used as the source databases for said applications.

Scenario 2: Data Migration

Data is housed in a deprecated or soon-to-be deprecated on-premise database and needs to be migrated to a new cloud database.

Scenario 3: Backup and Disaster Recovery

Data needs to be backed up for disaster recovery (high availability).

Suppose there are a million transactions per day. As a measure of prevention policy, the data needs to be backed up in a different data center in case of data loss in the main database. Qlik Replicate can be scheduled to create a copy of the Database every night after peak hours making use of its Full Load functionality.

If the backup needs to be in real-time, Qlik Replicate CDC capabilities can be used to send the data as soon as it’s committed in the source database (application database), creating a trusted and consistent copy of the main database during business hours instead of once a day.

Qlik Replicate can also work with some endpoint-specific capabilities such as SQL Server’s High Availability in case the company has multiple nodes in a cluster, making Qlik Replicate switch between the nodes in case of failover.

Qlik Enterprise Manager

Qlik Enterprise Manager provides a single point of control for designing, executing, and monitoring Qlik Replicate and Compose tasks. If your site has multiple Qlik servers multiple tasks, Enterprise Manager is able to change the design, management, and monitoring of these tasks.

In what scenarios can it be used?

Scenario 1: Managing Servers and Tasks

Managing several servers and tasks from a Qlik Replicate and Qlik Compose environment can pose significant effort.

Qlik Enterprise Manager is valuable as the single point of control and monitoring for all servers involved and provides granular permissions control for multiple users.

It is installed in a Windows machine and can be later accessed by client machines across the organization, enabling users to create, edit, delete and monitor both Qlik Replicate and Qlik Compose tasks. Multiple tasks can be started/stopped by selecting them and performing the action altogether instead of individually opening them via Qlik Replicate/Compose UI.

This can be further enhanced with the use of the REST API made available by Qlik Enterprise Manager, which enables the organization to programmatically manage the servers, being able to start/stop tasks, create new tasks, edit task endpoints and settings, retrieve monitoring information such as: changes applied, tables loaded, latency accumulated and much more making use of the embedded .NET, Python and REST API calls that Enterprise Manager provides.

Scenario 2: Resource Monitoring

Resource usage needs to be tracked across a company's Qlik Data Integration products, such as monitoring Memory usage, Disk space usage.

It also allows monitoring of Qlik Replicate task metics, such as: average throughput, number tables loaded, total applied changes, accumulated latency and more.

Example: With three different Qlik Replicate servers in three different countries, this can all be centralized and tracked by only one Enterprise Manager Analytics machine, which uses a PostgreSQL backend as the database to store all this information.

The PostgreSQL database can be installed in the Qlik Enterprise Manager machine itself (greater performance) or can be sitting in a separate machine in the same network (on-premise machine, VM).

Qlik Compose

Qlik Compose automates the costly and time-consuming process of designing, coding and continually updating your data warehouse and data marts. Qlik Compose helps to eliminate traditional costly ETL development resources while providing fast, agile delivery of information for Big Data Analytics and Business Intelligence (BI) projects.

In what scenarios can it be used?

Real-Life Scenario: Creating a Data Warehouse for the Pharmacy Industry

A pharmacy company needs to gather, centralize, and distribute data from a clinical trial to evaluate the safety and effectiveness of a new medication. The trials are held in different countries, being generated in different labs with a variety of systems that by nature are incompatible.

They created a single point of data storage (SQL Server on Azure), and with this, the company enabled all the employees to store the data that will later be organized and distributed by Qlik Compose.

After the trials are finished, all the data is gathered in the previously mentioned SQL Server on Azure. A BI architect working with Qlik Compose can then create a model that will cleanse the data by normalizing and applying needed transformations. Later, Qlik Compose will automate the designing of the warehouse, generating ETL code, and quickly applying updates, all whilst leveraging best practices and proven design patterns.

Later he’ll be able to set up a Data Mart using Qlik Compose so that the Datawarehouse would be made available for employees from different parts of the organization, from business managers to academic students so reports can be generated, as well as enabling ease of implementation of Data Analytics for the development and improvement of new products.

Qlik Catalog

Qlik Catalog is a modern enterprise data management solution that simplifies and speeds up the curation, preparation, and delivery of trustworthy, actionable data to business users across the enterprise.

In what scenarios can it be used?

Real-Life Scenario: Creating a Data Marketplace and Integrating with Qlik Sense Analytics

A company needs to onboard data to another Qlik product: Qlik Sense.

Before taking the data to Qlik Sense, employees from the company can “shop” for data, using an interface very similar to an e-commerce website. For example, the Purchasing team would be interested in selecting data from a specific product such as price, delivery time, availability in stock etc, while the Sales team would “shop” potential customer’s data such as region, sector, previous deals and more.

All of this can then be exported to Qlik analytics tools, such as Qlik Sense in which the departments can ask questions related to their own interests, such as for Sales Management: “How many products X did we sell while product Y was out of stock in Region Z?” or for Purchasing department: “What is the average number of days it takes Store X to sell all of their clothing products in stock? And usually how much do they spend on refilling said stock?”

Example of a pipeline using Qlik Data Integration products:

Qlik Data Integration Pipeline.png

In the simple example above, we can see the data flow of a use case in which the raw data is retrieved from a CRM using Oracle as a backend and storing Ads data in a PostgreSQL backend.

Using Qlik Replicate (managed by Enterprise Manager), we can load all this separate on-premises data into an Azure SQL database on the cloud.

Later, the data in this cloud database can be easily and quickly transformed and loaded into a Data Warehouse using Qlik Compose, which in turn can be used as a Catalog of data for the employees of the company and later used for Data Analytics using Qlik Sense, which would make the data that was raw in the beginning turn into valuable information so the company can take more data driven business decisions.