idea Re: Improve attrep_status performance on Snowflake as a target - Status changed to: Closed - Archived in Suggest an Idea

Improve attrep_status performance on Snowflake as a target

Prabodh — Wed, 18 May 2022 20:02:20 GMT

We are using Snowflake as a target and are replicating 800+ table from various sources using 75 tasks.

We have enabled attrep_status table and want to leverage this table for latency monitoring.

Replicate tasks 'updates' this table. Because of nature of Snowflake concurrent updates on a table fail. Snowflake has a default setting where it kills transactions if there are more than 20 transactions waiting on a table.

With 75 tables, there is a possibility that at a given point in time, more than 20 task try to update the table simultaneously. As a result these updates fail.

Because Replicate has a hard limitation of storing the control tables in the target DB, I would like Replicate to support 100's of tasks writing to Snowflake and be able to keep information in attrep_status consistent.

One option is to insert a new status in the table instead of update. Inserts do not create locks and hence would not have this issue.

From now on, please track this idea from the Ideation por...

Meghann_MacDonald — Wed, 02 Aug 2023 15:37:00 GMT

From now on, please track this idea from the Ideation portal.

Link to new idea

Meghann

NOTE: Upon clicking this link 2 tabs may open - please feel free to close the one with a login page. If you only see 1 tab with the login page, please try clicking this link first: Authenticate me! then try the link above again. Ensure pop-up blocker is off.

Re: Improve attrep_status performance on Snowflake as a target - Status changed to: Closed - Archived

Ideation — Wed, 02 Aug 2023 15:37:02 GMT