Qlik Replicate: Latency impacts from Target Apply stage switching to one-by-one mode

[Steve_Nguyen]

Qlik Replicate is able to process changes in near real-time when Batch Optimized Apply is enabled in the Task Settings > Change Processing Mode settings. This will commit changes in pre-grouped batches in an efficient way.  The alternative mode is Transactional Apply which will commit changes in the order they are committed, i.e sequentially or one-by-one.  Batching the changes dramatically improves task performance and reduces data latency.

In this article, we describe the latency impacts when change processing or Target Apply switches from batch apply mode to one-by-one mode.

When the table changes are divided into chunks or groups (of inserts, updates and deletes) for batch apply, it is possible for issues to occur, causing the task to switch to one-by-one:      

1. When a table is suspend table and it goes into suspended state
2. When there is an Apply conflict ( and when suspend) - go to suspend / stop task
3. If there is a target connection issue - the task will try the same bulk again

Then the next batch will be one-to-one and not related to last batch.

Below are additional cases where the task switches to one-by-one mode:

• If there is an update or delete and one of the pk is null
  
  
• If there is no Primary Key on a table you are trying to apply updates to.  You may see messages like:  
  
  [TARGET_APPLY ]V: Apply no Bulk Event
  [TARGET_APPLY ]T: Got non-Insert change for table 1 that has no PK. Going to move the Inserts in the current bulk to the list of changes for tables with no PK since their number did not reach the threshold.
  
  
• If source table definition or table definition has LOBs and those LOBs are supported and it is set to unlimited or a large value. These settings can be found in:
  1. Task Settings
  2. Metadata
  3. Replicate LOB columns
     
• If the change is an update and the primary key is changing on the source table
  
• If the change is an update and table is not found on the target or if the update contains new PK value not found on target table.
  
• If we exceeded the max column limit:  kAPR_ENDPOINT_CHANGE_TABLE_EXCEEDS_ROW_SIZE
  
• If target does not support bulk apply
  
• If there is an error and it's not a table error. For example, zero row effected "]E: 0 rows affected [122510] Zero rows affected" will break the bulk and increase latency.