topic Re: Resuming task using source sequence number in Qlik Replicate

Resuming task using source sequence number

MoeE — Mon, 15 Apr 2024 04:12:19 GMT

Hi,

I have a general question I'm wondering about. I saw in this article that you can resume or restart a task using the sequence number shown in the logs when a task stops. I understand that this is available only for certain endpoints.

https://community.qlik.com/t5/Official-Support-Articles/How-to-Start-Replicate-task-from-timestamp-or-bin-log-position/ta-p/1909623

So this appears to be the best "timestamp" to restart/resume a task without data loss or duplicates. However I am wondering if there is a possibility of the scenario where there could have been a long running uncommitted transaction that would be skipped if we restart/resume using this "timestamp". Is this possible or not? Please help my understanding.

Regards,

Mohammed

Re: Resuming task using source sequence number

DesmondWOO — Mon, 15 Apr 2024 07:14:24 GMT

Hi @MoeE ,

Yes, it is possible that some transactions may be lost. For example,

15:00 Begin Transaction
15:02 Insert Record
15:04 Commit

Let's say you resume task at 15:01. Replicate ignores the inserted record because it can't find the corresponding BEGIN transaction.

Regards,
Desmond

Re: Resuming task using source sequence number

john_wang — Mon, 15 Apr 2024 09:12:52 GMT

Hello @MoeE ,

Besides @DesmondWOO comments, I'd like to separate the 2 startup modes of

>> restart/resume using this "timestamp"

1- there is significant difference between RESUME and restart from "timestamp"

RESUME: Qlik Replicate will load the .TSWP files from disk to memory and continue processing the 'in doubt' transactions in Desmond's sample. This is because Qlik Replicate will store the 'uncompleted' transaction into .TSWP files in previous task stop.

However, if the task is restarted from timestamp, then no .TSWP file will be handled and the 'in doubt' transactions will be ignored.

2- Desmond's data missing scenario occurs for some DB only, eg Oracle source.

Hope this helps.

John.

Re: Resuming task using source sequence number

MoeE — Mon, 15 Apr 2024 23:24:36 GMT

Hi @john_wang and @DesmondWOO ,

Thank you for the help.

Regards,

Mohammed

Re: Resuming task using source sequence number

john_wang — Tue, 16 Apr 2024 00:34:50 GMT

Thank you for your great support Mohammed! @MoeE