Qlik Replicate: Bulk CDC to Snowflake changes to one-by-one mode

[shashi_holla]

There are many scenarios when Bulk CDC to Snowflake can switch to one-by-one mode and cause latency. This article is to highlight possible scenarios.

Primary Key Value change

A Primary Key value change on the source. This can be identified in the Qlik Replicate task logs by locating the [TARGET_APPLY] line. 

Example:

[TARGET_APPLY ]T: Update on table 1 changes PK to a PK that was previously updated in the same bulk

This is not an ideal scenario, and we need to ensure PK column value doesn't get modified on the source. If the design cannot be changed then this table needs to be in a separate task to ensure it doesn't affect bulk processing of other tables in a task.

Duplicate Inserts and Updates to the target

This can be fixed by setting UPSERT mode in Task Settings.

1. Open the task
2. Open the Task Settings
3. Expand Error Handling
4. Open Apply Conflicts
   
   
   
   
5. Set:
   • Duplicate key when applying INSERT: to UPDATE the existing target record
   • No record found for applying an UPDATE: to INSERT the missing target record

This setting works even if there are LOB data starting with Qlik Replicate 2021.11 SP06/2022.05 SP02 and higher.

Environment

Qlik Replicate