Qlik Replicate Full Load and CDC Split Task: Considerations

Michael_Litz — Wed, 04 Jan 2023 09:26:36 GMT

Note: Having a task split into two tasks, one for Full Load and the other for CDC is not the usual configuration.

Why you may want to split a task:

A full load may take too long to complete to be practical.
Other times many full load tasks are run at the same time using different ranges on each.

Full Load and CDC Combined Task:

During a Full Load and CDC task combined, the CDC actually starts before the full load and caches changes read from the source transaction log. As the full load completes, the task is able to apply these cached changes in a way that does not produce errors such as duplicate keys for insert and records not found for the update.

Full Load and CDC Split Task:

In a split task environment, these are some considerations:

If you run the CDC task while the Full load task is starting it could possibly run into the above-mentioned errors.
If you don't start the CDC with a timestamp from before the Full Load started, then you could possibly miss some transactions.
One way to mitigate the above two issues would be to set error handling / apply conflicts to upsert/merge setting which will basically turn updates to a non-existing record into an insert and an insert of a record that already exists into an "Update"

Please reference the KB article: Qlik Replicate: Implementing UPSERT and MERGE modes by applying a Conflicts Handling Policy

The information in this article is provided as-is and to be used at own discretion. Depending on tool(s) used, customization(s), and/or other factors ongoing support on the solution below may not be provided by Qlik Support.

Environment

Qlik Replicate

article Qlik Replicate Full Load and CDC Split Task: Considerations in Official Support Articles

Qlik Replicate Full Load and CDC Split Task: Considerations

Why you may want to split a task:

Full Load and CDC Combined Task:

Full Load and CDC Split Task:

Environment