Confirmation of Understanding Regarding Latency and Parallel Processing.
Hello Support Team,
I hope you are doing well.
Could you please review the following questions and confirm if my understanding of the system behavior is correct?
▼Questions 1.In a Log Stream Staging task, what is the difference between Total_Latency and Source_Latency?
2.In a Replication task, what is the difference between Total_Latency and Source_Latency?
3.Using the built-in notification features, we set a latency alert threshold of 1 hour (3600 seconds), and alert emails are triggered occasionally. Recently, we received alerts at almost the same time for a paired set: the Log Stream task reported a latency of 3604, while the Replication task reported 3603. In this scenario, what is the "actual" latency (i.e., the total time elapsed since the transaction was committed on the source side)? Is it the sum of the two values (7207 seconds), or should we only focus on the higher value? Or, since the tasks are paired, is the actual latency effectively the same, making the choice of which to monitor arbitrary? (Note: Latency alerts always occur as a set for both paired tasks, and in almost all cases, the latency values for the Log Stream and Replication tasks are nearly identical.)
4.While increasing parallelism seems like a viable method to improve latency, is it true that parallel processing is limited to Full Load? Is parallel processing unavailable for Apply Changes?
▼My Understanding 1.Log Stream Task: Source_Latency: The time from when a transaction is committed in the source DB until Qlik Replicate captures that update. Total_Latency: The time difference from the commit in the source DB until the data is successfully written to the Qlik Replicate server (staging folder).
2.Replication Task: Source_Latency: The time elapsed until the task reads the data from the Log Stream staging folder. Total_Latency: The total time gap from the commit in the source DB until the data is applied and visible in the target DB.
3.Combined Latency: When using Log Stream, I believe the sum of the latencies from the Log Stream Staging task and the Replication task represents the total time it takes for data committed in the source DB to be applied to the target DB. Is this recognition correct?
4.Parallelism in Apply Changes: Aside from splitting a single task into multiple Replication tasks to achieve parallel processing for Apply Changes, are there any other methods to enable parallel processing during the change application phase?
In the User Guide, the terms Source Latency, Target Latency, and Overall Latency are defined. In the PERFORMANCE logger, however, the metrics reported are source latency, handling latency, and target latency. I believe the mappings should be as follows
User Guide
Performance Logger
Source Latency
Source Latency
Target Latency
Handling Latency
Overall Latency
Target Latency
I believe the terminology used in the User Guide is clearer and more precise. However, the application still reflects the original terms defined by R&D. If needed, you may open a support ticket to confirm this.
Regarding the other latency questions, it may be easier to understand if you apply the following equation:
This means that latency occurs at the source endpoint. In general, target latency should remain very low because Qlik Replicate writes changes directly to disk.
This calculation is the same for both staging task and replication task.
Q3 Currently, notifications do not provide detailed latency information. If you need more detail, you can set the TRACE level for the PERFORMANCE logger. This will allow you to find source latency, handling latency, and target latency in the task log
Q4 Yes, Parallel Load is supported only for the Full Load phase, because CDC involves DML changes (insert, update, delete) that must be applied in the exact order to maintain consistency. If you mean running CDC across multiple tables simultaneously, you can create multiple replication tasks, but please note that this approach will increase I/O overhead.
Regards, Desmond
Help users find answers! Do not forget to mark a solution that worked for you! If already marked, give it a thumbs up!
I apologize for the confusion. Total latency was meant to refer to Overall latency.
Q3
Is it correct to understand that this varies depending on the situation?
For example, if the situation were as follows, how would we calculate the time taken from when the update is confirmed at the Source Endpoint until the data is written to the Target Endpoint and confirmed?
When source latency occurs, it represents the time taken for reading from the Source Endpoint
When target latency occurs, it represents the time taken for writing to the Log Stream Staging folder
For Replication Task:
When source latency occurs, it represents the time taken for reading from the Log Stream Staging folder
When target latency occurs, it represents the time taken for writing to the Target Endpoint
Therefore, my understanding is that simply adding the Overall Latency of Log Stream Staging Task and Replication Task would not equal the total time taken from when the update is confirmed at the Source Endpoint until the data is written to the Target Endpoint and confirmed.
Q4
Thank you for your clear explanation. That helps me a lot.
In the User Guide, the terms Source Latency, Target Latency, and Overall Latency are defined. In the PERFORMANCE logger, however, the metrics reported are source latency, handling latency, and target latency. I believe the mappings should be as follows
User Guide
Performance Logger
Source Latency
Source Latency
Target Latency
Handling Latency
Overall Latency
Target Latency
I believe the terminology used in the User Guide is clearer and more precise. However, the application still reflects the original terms defined by R&D. If needed, you may open a support ticket to confirm this.
Regarding the other latency questions, it may be easier to understand if you apply the following equation:
This means that latency occurs at the source endpoint. In general, target latency should remain very low because Qlik Replicate writes changes directly to disk.
