Latency to high after restarting task

[Jperezmatus]

Reasuming a task just after stoping it is causing the latency to increase up to 30 minutes aproximately.  This is happening after we moved the target to azure cloud.  Is there a way to tune this?

Thank You

[Jperezmatus]

Hi Bill, this settings are, practically, the default settings.

BATCH Tunning

  Apply batched changes in intervals longer than 1 sec, but less than 30 sec, force apply a batch when processing memory exceeds 500 MB.

Apply Batched changes to multiple tablets concurrently is OFF

offload transactions in progress to disk if Total transaction memory size excceds 1024 MB, transactions duration excceds 60 sec.

Statements cache 50.

Thank  You

[Steve_Nguyen]

performance tuning is normally done by our Professional service team, if you need further performance tuning, best to engage with PS

[Jperezmatus]

Hi @narendersarva , we don't have apply latency problem while the task is runnig.  I have seen 20,000 incoming changes with an apply latency really low.  The issue is only when restarting and reasuming the task. I don't know if this is the expected beaiviour.  Even with 0 pending transaction , when reasuming the task it begins to acummulate incoming changes for up to 40 minutes. 

Thank You

[Jperezmatus]

Hi @Bill_Steinagle , we practically have default setting for change processing.

Apply batched changes in intervales  longer than 1 sec, but less than 30 sec. force apply a batch processing memory exceeds 500 MB. 

Apply batched changes to multiple tables concurrently is OFF

Offload transactions in progress to disk if total transactions memory size exceeds 1024 MB, transactions duration exceeds 60 sec.

Statement cache size 50 statements

Regards

[Bill_Steinagle]

Hi,

I would suggest you open a ticket with Support so we can address the issue in the lab. We would setup the scenario in the lab to confirm the use case before we have a JIRA created. 

Thank you!

[Heinvandenheuvel · Accepted solution]

Your UI snapshots suggest that after a resume any fresh incoming changes found are quickly read and pushed to the apply stream where they sit in memory for a while before migrating to and on-disk storage due to failing to apply in time.

I suspect the system is failing to co0nnect to the target DB. This is a SOFT failure in Replicate. It will just keep on trying. Check the logs for the lines immediately following '[TARGET_APPLY ]I: Going to connect to server  ' Eventually it should be followed by '[TASK_MANAGER ]I: All stream components were initialized '

You probably want to and to stop, switch logging to TARGET_APPLY TRACE an resume again. Hopefully any connection failure has enough details. f not, do it again with logging to VERBOSE.

Sometimes initial connections, due to DNS issues perhaps, seem to go around the world a couple of times, or to the moon and back before finding the target. Once connected all will might fine. 

If the connections do succeed in reasonable time, then look at the 'apply'  events and the log entries for that.

Also, The tasks is not Re-loading tables as part of the resume is it now? Perhaps it is convinced the target table structure changed?

Hein.

[Jperezmatus · Accepted solution]

Hi @Heinvandenheuvel  , I have set the log to verbose.  I see the task is creating all the metadata again for every table and , apparently, this is why the latency increases.  I guess this the expected behaviour, and that is why every restart takes about the same amount of time.

Thank You