Qlik Replicate Loading Best Practices

Troy_Raney — Wed, 01 Feb 2023 06:58:56 GMT

This Techspert Talks session covers:

Full Load vs. Reload
Examining Task Options
Configuration Best Practices

Environment:

Qlik Replicate

00:00 – Intro

00:58 - Full Load tasks

02:26 - Adjusting load priority

04:13 - Loading LOB fields

05:43 - Full Load Passthrough Filter

09:58 - Using Insert Date and Update Date

13:42 - Parallel Loading

15:28 - Setting up Notifications

18:23 - Q&A: Can Replicate do compressions?

18:45 - Q&A: What can affect performance?

20:03 - Q&A: How does metadata affect performance?

20:19 - Q&A: Is Replicate compatible with Oracle Data Bricks?

21:09 - Q&A: What if a task gets "Max Row Size Limit"?

21:58 - Q&A: Are Oracle OCI Plugable Databases supported?

22:39 - Q&A: What is the recommended version of Qlik Replicate?
23:59 - Q&A: How to limi or filter notifications?

24:58 - Q&A: How can you restart a stopped task?

25:49 - Q&A: Is Qlik Replicate free with Qlik Cloud?

26:11 - Q&A: How to change the Primary Key field?

27:21 - Q&A: How to investigate task latency?

28:15 - Q&A: What kind of hardware helps performance?

28:47 - Q&A: How to adjust segments to improve load performance?

Q&A:

Q: If I have 10 tables in a task, could I select only 1 table for a full load or does it have to be managed via a separate task?

A: Reload of a specific table is possible from the monitor screen. The whole task/all tables will reload at the task level.

Q: What is the best way to debug the CDC apply changes process?

A: Review the Troubleshooting Qlik Replicate Latency with Kelly Hobbs. She details this process.

Q: Do lookups affect latency of tables?

A: Yes, there is some overhead when the task issues the select statement on the source in order to get the data.

Q: If our target database is used by another development, will it be affected? Is there any possibility of data deadlock?

A: Yes, if the other application ever issues a lock on the target table rows that could cause an error for the Replicate task.

Q: Is there a way to setup a Notification that sends only a few e-mails when a Task has an Error or Task Stops because it can't connect to the Target and needs to stop? My experience has been that it fills my e-mail inbox with hundreds of pages.
A: You can set up rules in your mail, or send notifications to the event viewer and have a program that scans the event viewer and sends emails based on the logic coded into the program.

Q: What happens in Replicate on a delete?

A: The delete will remove the record from the target. We do have a way to do a soft delete that will keep the record and mark it as deleted. Check out this article: Filter Deletes in Replicate

Q: The options available for tuning LOB handling when we use Log Stream as source as opposed to native application type source endpoint, are limited. So is there any way to explicitly pass the config options in task JSON configuration?

A: The LOB limitations on the LOG stream are by design and cannot be overridden.

Resources:

Customizing Data with Qlik Replicate

Adding Update Date field

Adding Table level Insert Date

Qlik Replicate Supported End Points

Troubleshooting Qlik Replicate Latency

An Introduction to Qlik Replicate Tasks: Full Load vs CDC

Click here for video transcript

article Qlik Replicate Loading Best Practices in Official Support Articles

Qlik Replicate Loading Best Practices