We are using Snowflake as a target and are replicating 800+ table from various sources using 75 tasks.
We have enabled attrep_status table and want to leverage this table for latency monitoring.
Replicate tasks 'updates' this table. Because of nature of Snowflake concurrent updates on a table fail. Snowflake has a default setting where it kills transactions if there are more than 20 transactions waiting on a table.
With 75 tables, there is a possibility that at a given point in time, more than 20 task try to update the table simultaneously. As a result these updates fail.
Because Replicate has a hard limitation of storing the control tables in the target DB, I would like Replicate to support 100's of tasks writing to Snowflake and be able to keep information in attrep_status consistent.
One option is to insert a new status in the table instead of update. Inserts do not create locks and hence would not have this issue.
