Finding Number of pending updates when job is stopped

[adbdkb]

Hi,

From my understanding, when we stop a  running task  and  then later Run the Task with Resume Processing, it would start from the timestamp when it was stopped, right?

One question is - Is there a way to find out how many updates are pending when we stop the task? 

Thank you

[Alan_Wang]

Hi @adbdkb 

When a task is stopped, Replicate will keep track of the last transaction replicated to the target. This is the savepoint that Replicate will start from, not the timestamp of when the task was stopped. These save points are saved as stream positions for the product which is also listed in the task logs.

The amount of changes pending when the task is stopped is shown in the Replicate UI. You will see the number of changes still stored on the disk. These changes have not been replicated yet and will continue once the task is resumed.

[adbdkb]

Thank you.  Where would I see it in the UI?

Is it the "Incoming Changes" part under the Change Processing Tab? What are the In Memory and On Disk bars for source and target below?

[Alan_Wang · Accepted solution]

Hi @adbdkb 

The changes would show up on the "On disk" bars. In-memory bars are changes being currently applied to the target, but as you stop the task, Replicate will attempt to finish the apply process within 30 minutes and then move any leftover changes to the "On disk" section as well. Incoming changes are transactions being read from the source, which will soon move to the bars below depending on their status.

More information about this UI can be found in this link

https://help.qlik.com/en-US/replicate/May2022/Content/Global_Common/Content/SharedEMReplicate/Monito...

[adbdkb]

Hi @Alan_Wang  - Need help with one more part.  When the Task is in Stopped state, I should see the pending changes in the currently hi-lit in blue part.  At least, that is what I understood from your earlier response.   Is that understanding correct?

The reason I am asking this is that - I had created another task with the same source endpoint and had kept that task running whereas the first task is in stopped state. 

The one that is running shows changes in the Change Processing tab for this morning, and I had expected to see that number under the incoming changes for the stopped task.

Have I understood how the stopped task and incoming changes part work, correctly?

 

Thank you

[Alan_Wang]

Hi @adbdkb 

The pending changes from a stopped task would be in the 4th bar of the bottom row. They are changes waiting to be applied but are still stuck on the Replicate server. The highlighted incoming changes are transactions being read, but they will be moved to the 4 bars below as Replicate stores and applies them to the target. 

[adbdkb]

Thanks.  But, then why am I not seeing it in the above image?  Or will I see it only when I start the task again? 

[Alan_Wang]

Hi @adbdkb 

In the picture above, there are no pending transactions. Everything read and captured by Replicate has been transferred to the target and the task is update to date with the replication process. You only really see pending transactions if the Replicate server is not able to keep up with applying the number of transactions being read. 

Ex) Replicate is reading 100k changes per minute but is only able to apply 80k per minute. Once the task is stopped, Replicate will stop reading the source for changes but will continue to apply changes to the target for up to 30 minutes.

You won't see pending transactions on a stopped task unless you have a very large amount of transactions, usually millions.

[adbdkb]

Thanks for the explanation.  My question about stopped task and seeing pending tasks is related to the below situation.

I have stopped the task for 2-3 days now, so the finishing part is not in play.  I have another task reading the same source that I have kept running.  So, in the last 2-3 days there are about 6000 updates in the source.  So, since the task has been kept stopped, I had thought that it will show up as pending changes. 

Is that not correct?

Thanks

[Alan_Wang · Accepted solution]

Hi @adbdkb 

When a task is stopped, it no longer captures changes. If the source had 6000 updates during the time frame that the task is stopped, the task will not read it until it is started back up again. The task will keep track of the transaction position of where it stopped for the source database and continue reading from that position when it is started again.

No changes are lost, it is just behind schedule for the replication process. Note that if you left the task in a stopped state for days and millions of changes need to be read, you will possibly see latency for the task and a pile-up of pending changes.

Each task works independently, even if they share the same source endpoints. All the info for each task remains separate. Incoming changes, pending changes, latency etc. are all related to the single task that it is on.