Best approach for large, long running transactions in Qlik Replicate
Hi Support,
In a scenario where it's known that a high volume, long running transaction is about to be committed in the source database, what approach should be taken in Replicate?
Should Replicate be kept running as per usual? Or is it better to have the task stopped whilst the transaction is open in the source and only resumed once the transaction has been committed in the source database? Or is there no difference between the 2 approaches.
Short answer: The task should be kept running as normal.
The main objective is to allow Qlik Replicate to catch up with changes as quickly as possible. In most RDBMS platforms, Replicate can only capture committed changes, since changes become visible only after the transaction is committed. In some databases (e.g., Oracle and IBM DB2 LUW), Replicate can start reading changes from the logs even before the transaction commits.
By keeping the task running, Replicate can begin reading changes as soon as they are written to the logs. If the large transaction is eventually committed, the changes will be applied to the target. If it is rolled back, the changes will simply be discarded.
Pros of keeping the task running
Replicate continues reading logs without interruption
No restart or repositioning overhead
Faster catch-up once the transaction is committed
Comparison to stopping/resuming the task
Replicate stops reading logs, which may slightly reduce load on the source system
However, it introduces restart overhead and delays in catching up
Additional operational complexity and potential risks during resume
In summary, keeping the task running is generally the preferred and safer approach.
Hope this helps.
John.
Help users find answers! Do not forget to mark a solution that worked for you! If already marked, give it a thumbs up!
Short answer: The task should be kept running as normal.
The main objective is to allow Qlik Replicate to catch up with changes as quickly as possible. In most RDBMS platforms, Replicate can only capture committed changes, since changes become visible only after the transaction is committed. In some databases (e.g., Oracle and IBM DB2 LUW), Replicate can start reading changes from the logs even before the transaction commits.
By keeping the task running, Replicate can begin reading changes as soon as they are written to the logs. If the large transaction is eventually committed, the changes will be applied to the target. If it is rolled back, the changes will simply be discarded.
Pros of keeping the task running
Replicate continues reading logs without interruption
No restart or repositioning overhead
Faster catch-up once the transaction is committed
Comparison to stopping/resuming the task
Replicate stops reading logs, which may slightly reduce load on the source system
However, it introduces restart overhead and delays in catching up
Additional operational complexity and potential risks during resume
In summary, keeping the task running is generally the preferred and safer approach.
Hope this helps.
John.
Help users find answers! Do not forget to mark a solution that worked for you! If already marked, give it a thumbs up!
What is the source endpoint? Unless the large transaction is causing significant resource constraints (e.g. CPU, Memory, I/O), I recommend letting the task keep running.
Regards, Desmond
Help users find answers! Do not forget to mark a solution that worked for you! If already marked, give it a thumbs up!
