For general advice on how to troubleshoot Qlik Replicate latency issues, see Troubleshooting Qlik Replicate Latency and Performance Issues.
If your task shows latency issues, one of the first things to do is to set the logging component performance to trace and run the task you identified for five to 10 minutes and review the resulting task log.
We advise you to:
- Open the task log in a text editor (such as Notepad++),
- then search source latency
- and choose Find all in current document.
This will list all available latency information. We can now identify a trend.
Remember, Target latency = Source latency + Handling latency.
No latency
[PERFORMANCE ]T: Source latency 0.00 seconds, Target latency 0.00 seconds, Handling latency 0.00 seconds (replicationtask.c:3703)
The source, target, and handling latency are all at 0 seconds.
Source latency
[PERFORMANCE ]T: Source latency 7634.89 seconds, Target latency 7634.89 seconds, Handling latency 0.00 seconds (replicationtask.c:3793)
[PERFORMANCE ]T: Source latency 7663.00 seconds, Target latency 7663.00 seconds, Handling latency 0.00 seconds (replicationtask.c:3793)
[PERFORMANCE ]T: Source latency 7690.12 seconds, Target latency 7693.12 seconds, Handling latency 3.00 seconds (replicationtask.c:3793)
[PERFORMANCE ]T: Source latency 7710.25 seconds, Target latency 7723.25 seconds, Handling latency 13.00 seconds (replicationtask.c:3793)
The source latency is higher than the handling latency. The key point is to look at handling latency, it must be lower than the source latency.
Cause:
- Source database busy
- Lack of network bandwidth to the source
- There are a lot of CDC changes to capture (the task has stopped for some time and only just resumed)
- There are a lot of CDC changes to capture (the source database has maintenance or a sudden influx of DML operations occurred)
If the source latency decreases during your monitoring, it is a good sign that the latency will recover; if it increases, review the causes mentioned above and resolve any outstanding source issues. You will want to consider reloading the task.
Target latency
[PERFORMANCE ]T: Source latency 2.05 seconds, Target latency 7116.05 seconds, Handling latency 7114.00 seconds (replicationtask.c:3793)
[PERFORMANCE ]T: Source latency 2.77 seconds, Target latency 7150.77 seconds, Handling latency 7148.00 seconds (replicationtask.c:3793)
[PERFORMANCE ]T: Source latency 2.16 seconds, Target latency 7182.16 seconds, Handling latency 7180.00 seconds (replicationtask.c:3793)
The target latency is higher than the source latency.
Cause:
- Target is busy
- Lack of network bandwidth to the target
- A bottleneck in the target database computing power, especially for cloud endpoints
If the target latency continues to increase, consider reloading the task.
Handling latency
Identifying whether or not you are looking at handling latency or target latency can be tricky. When the task has target latency, the queue is blocked, so the handling latency will automatically be higher as well (remember: Target latency = Source latency + Handling latency).
The key point to decide if it is handling latency is to check if there are a lot of swap files saved in the sorter folder inside the task folder of the Qlik Replicate server.
In addition, if the task log shows when the task is resumed, the handling latency increases dramatically from 0 seconds (or a low number) to a much higher value in a very short time. This can then be clearly identified as a handling latency:
2023-05-10T08:21:02:537595 [PERFORMANCE ]T: Source latency 5.54 seconds, Target latency 5.54 seconds, Handling latency 0.00 seconds (replicationtask.c:3788)
2023-05-10T08:21:32:610230 [PERFORMANCE ]T: Source latency 4.61 seconds, Target latency 55363.61 seconds, Handling latency 55359.00 seconds (replicationtask.c:3788)
This log shows handling latency increased from 0 seconds to 55359 seconds after only 30 seconds of a task's runtime. This is because Qlik Replicate will read all the swap files into memory when the task is resumed. In this situation, you need to reload the task or resume the task from a timestamp or stream position.
Related Content
Environment
