How To Get Started with Qlik Replicate

[KellyHobson]

This is a guide to get you started working with Qlik Replicate. 

• System Requirements
• Downloading the software
• Installation steps
• Upgrade steps
• Architecture
• Getting started: Set up a task
• Helpful Videos
• Task and Task Settings
• Filters and Transformations
• Monitoring Task
• Log Stream
• Latency
• Debugging Steps
• Contacting Support or Professional Services
• Optional White Papers
• Environment

System Requirements

Below contain information about the requirements for a successful Qlik Replicate deployment and the supported platforms and endpoints. 

Overview in Replicate User Guide
Supported Platforms and Endpoints

For complex systems or configurations, we recommend utilizing Qlik Professional Services.

Downloading the software

Qlik Replicate major releases are named by the Year.Month format (example May 2022 release is v2022.5).

1. To access the Download Page, click here. 
2. From there select:
   1. Data Integration
   2. Qlik Replicate
3. Choose your version and download it.

Installation steps

The Qlik Replicate user guide has step-by-step instructions for installing and deploying Qlik Replicate in your environment. 

Windows 
Linux
Linux installation walk through

Upgrade steps

Qlik Replicate releases software on a regular cadence between major releases to address bug fixes and product enhancements.

Upgrade steps can be found here.

 

Architecture

Qlik Replicate accelerates data replication, ingestion and streaming across a wide variety of heterogeneous databases, data warehouses, and Big Data platforms. After loading the selected tables to the target, Qlik Replicate’s high-performance change data capture (CDC) technology remotely scans transaction logs and rapidly delivers real-time data updates.

 

Getting started: Set up a task

Qlik Replicate change data capture (CDC) is log-based replication. Thus, it only changes data read from transaction logs and reduces the impact on the source database.

You can find two modes of Replication:

1. Full load:
   Automatically creates and loads the target
   
   
2. Change Processing (CDC):
   Captures changes in the source as they occur and applies to the target in near real-time

Helpful Videos 

Helpful YouTube videos to get an idea of how to set up a Qlik Replicate task

Simple Demo
Demo with MySQL source, Postgres target 
Demo with DB2 source, Oracle target

Task and Task Settings

When creating and managing your tasks, The main steps to set up your tasks are:

1. Create your source and target endpoints first. Replicate console > Manage Endpoints
   
   
   
   
2. Create the New task and drag and drop the Source and target endpoints onto the Designer window.
   
   Qlik Replicate Console > New Task
   
   
   
   
   
   
3. Select the table(s) that you wish to Replicate
   
   
   
   
   
   
4. Optional you can modify the name of the Schema and/or Table name when it is replicated to the target.
   
   
   
   
5. Choose Full Load if you wish to get a copy of the entire table to the target. More information about Full Load Settings 
   
   
   
   
6. Choose Change Processing for CDC changes to me moved to the target. More information about CDC Settings
   

   

7. Control Tables. These contain further metadata about the task and are written to the target. Apply exceptions is the only control table enabled by default.  More information in the user guide or this community article. 
   
   
   
   
8. Error Handling. Task Settings > Error Handling.
   
   This provides control over how you would like Qlik Replicate to handle data, environmental, or table errors and/ apply conflicts.  More detail from the user guide and here is a very helpful overview of how to implement UPSERT mode. 

Filters and Transformations

Filters and Transformations can be found in:

1. Task Designer
2. Table Settings
   1. Filters or Transformations (for task level) or
   2. Task Designer, Global Rules (for all tasks). 

The expression builder is handy for testing filters on your columns. 

We have prepared an example of filters for full load and CDC task and an awesome video about renaming tables, adding update date fields, table level transformations and more!

Here is another example for implementing filters.

Monitoring Task 

The monitoring tab provides real-time information about the status of Full Load or CDC jobs. This is a helpful article about how to interpret the info on the Monitor tab. 

Log Stream 

Log Stream task enables a dedicated Replicate task to save data changes from the transaction log of a single source database and apply them to multiple targets, thereby eliminating the overhead of reading the logs for each target separately.

Background in User Guide
Log Stream Setup 
Log Stream the Why and the How

Latency

A common request is to help tasks experiencing latency issues.  

Source Latency is the gap in seconds between when the source database wrote an event to its transaction log and when Replicate captured that change.

Apply Latency is the gap in seconds between capturing a change in one of the source tables and applying that change to the target endpoint.

Target Latency is the gap between when a commit is seen by Replicate (reading the source transaction log) and when the changes of that commit are seen in the target.

• Troubleshooting Latency and Performance Issues 
• Latency can be helped by Tuning your task.

Debugging Steps 

We love working with you but here are steps to enable you to diagnose and research issues on your own.

How to download a Replicate diagnostic package  
Replicate Knowledge Base for technical articles 
Replicate Forum for questions and discussions 

         Techspert talk from one of our experts on troubleshooting

          How to analyze a Qlik Replicate Log

Contacting Support or Professional Services

If you are still stuck and need some help, please reach out to the Qlik Support Team or Qlik Professional Services.

Which one to contact?
Support
Professional Services 

Optional White Papers 

These require you to register but are good marketing material if needed. 

High Performance Data Replication
ABB SAP Data to Snowflake

 

Environment

• Qlik Replicate 

The information in this article is provided as-is and to be used at own discretion. Depending on tool(s) used, customization(s), and/or other factors ongoing support on the solution below may not be provided by Qlik Support.