Qlik Replicate: ADLS Target High Target Latency

[DesmondWOO]

High target latency is observed when replicating to an Azure Data Lake Storage (ADLS) target.

Because ADLS is a file-based storage system rather than a traditional relational database, such high latency is unexpected.

Additionally, even though the File size reaches (KB) and Elapsed time reaches (Sec) thresholds are properly configured under the ADLS endpoint's Change Processing settings, Qlik Replicate fails to upload or transfer any CSV files to the target.

This behavior makes the replication task appear completely stuck or frozen.

Resolution

To resolve this issue, enable the internal parameter splitTransactionOverFiles on the endpoint. This forces Qlik Replicate to split large single transactions across multiple smaller files, allowing your file size and elapsed time thresholds to trigger properly and restoring normal file delivery.

1. Open the ADLS endpoint settings and switch to the Advanced tab
2. Click Internal Parameters
3. Enter the parameter splitTransactionOverFiles 

   The field is case-sensitive.

4. Enable the parameter
5. Save

Cause

By default, Qlik Replicate writes an entire source transaction into a single target file.

If the source system executes a massive transaction (for example, updating 1 million records in a single commit), Qlik Replicate must process and write all 1 million records into the same CSV file and upload it.

This causes the file transfer to hold indefinitely until the entire transaction is written, resulting in high latency.

Environment

• Qlik Replicate