topic Re: CDC Latency after DST Off in Qlik Replicate

CDC Latency after DST Off

brendanodwyer — Fri, 10 Nov 2023 03:56:20 GMT

We are replicating an Oracle Source and started to have latency issues after the time change (DST off).

Each replication task from our oracle source is setup with a Log Stream task in front of it that the task is reading from.

Since the time change we have been consistently behind by an hour in our CDC replication for these tasks. I looked in the logs and when the logstream task turned on we can see that the log stream tasks registered the time from oracle as below:

00002032: 2023-10-26T19:11:36 [SOURCE_CAPTURE ]T: SESSIONTIMEZONE: '-05:00' (oracle_endpoint_capture.c:142)

00002032: 2023-10-26T19:11:36 [SOURCE_CAPTURE ]T: SYSTIMESTAMP of DBTIMEZONE: '-05:00' (oracle_endpoint_capture.c:172)

00002032: 2023-10-26T19:11:36 [SOURCE_CAPTURE ]T: DBTIMEZONE: '-05:00' (oracle_endpoint_capture.c:236)

We had this issue in our PROD environment and one of our Non-PROD environments, but our Non-Prod Environment which is configured identical to our PROD environment seems to have fixed itself. I'm not sure how or why the latency decreased in our Non-Prod environment, but would like to understand and resolve how we can get the latency to decrease in our PROD environment.

Re: CDC Latency after DST Off

fbp — Fri, 10 Nov 2023 09:03:53 GMT

Hello

Do you see a steady latency? or a growing one?

Regards

Re: CDC Latency after DST Off

Steve_Nguyen — Fri, 10 Nov 2023 14:31:43 GMT

@brendanodwyer

you can check below article:

https://community.qlik.com/t5/Official-Support-Articles/Qlik-Replicate-Daylight-Savings-Guidance/ta-p/2041124

also, try to stop task , restart the Replicate service. and then resume task.

Re: CDC Latency after DST Off

brendanodwyer — Fri, 10 Nov 2023 14:53:43 GMT

@fbp our normal latency is between 30s and 5m for this source, and the latency now is 1h + our normal latency and its not increasing.

@Steve_Nguyen i've tried just turning off/on one of the replicate tasks but that doesn't fix it.

Our Non-Prod latency did come back this morning, and i stopped and started the LogStream task and the Replicate Task that reads from it. That seems to have fixed it and in the logs for the logstream task it shows that it has picked up the new timestamp/timezone from Oracle:

00006900: 2023-11-10T08:49:21 [SOURCE_CAPTURE ]T: SESSIONTIMEZONE: '-06:00' (oracle_endpoint_capture.c:142)

00006900: 2023-11-10T08:49:21 [SOURCE_CAPTURE ]T: SYSTIMESTAMP of DBTIMEZONE: '-06:00' (oracle_endpoint_capture.c:172)

Re: CDC Latency after DST Off

Heinvandenheuvel — Sat, 11 Nov 2023 02:45:54 GMT

I suspect there was never an actual latency, just a latency reporting issues due to timezone changes not being in sync between the various components in the solution. The reported latency may have been wrong , but the speed of replication 99% sure was not impacted at all.

Re: CDC Latency after DST Off

ferrel63 — Tue, 14 Nov 2023 15:08:18 GMT

Wow. We have been just throwing alike-named tables in the same subs. It's difficult to keep track of over 100 subs and what is inside of them. We do keep massive tables in their own subs, but didn't even think to have time-critical tables in their own subs. Really smart.

Re: CDC Latency after DST Off

Steve_Nguyen — Tue, 14 Nov 2023 16:59:05 GMT

@ferrel63 are you talking on the same topic of latency ?

Re: CDC Latency after DST Off

Heinvandenheuvel — Tue, 14 Nov 2023 21:00:26 GMT

@ferrel63 >>> Wow. We have been just throwing alike-named tables in the same subs. It's difficult to keep track of over 100 subs and what is inside of them. We do keep massive tables in their own subs, but didn't even think to have time-critical tables in their own subs. Really smart.

English please ?! What is a 'sub'? Please ask a colleague to read your question and help explain to the world.

Regards,

Hein.