What is the expected behaviour when a task breaks and enters one-by-one mode?
Hi,
Customer has a CDC task and has come across the issue that when one of the tables enters one-by-one mode, a lot of changes begin getting cached on disk and only 1 table is applying changes to the target while the rest of the tables are not applying any changes to the target.
I'd like to confirm whether this behavior is by design. From my understanding of the situation, this is what is happening. When a table experiences an issue that causes it to enter 1 by 1 mode. The changes from the rest of the table will be captured but won't be applied until this table has applied its changes and the task switches back to bulk apply processing mode.
We have identified that this table does not have primary key which is likely reason as to why it keeps breaking into 1 by 1 mode and have already suggested to separate tables without primary keys into another task.
Can someone please confirm that this is the normal behavior and to avoid the other tables queuing up in cdc, they must be placed into separate task from tables without primary keys. Thank you, I appreciate the help guys.
This is normal behavior. When Replicate encounters an issue, such as a duplicate record during the processing of a batch of records, it switches to a one-by-one mode. This mode allows changes to be applied individually, ensuring that valid records can still be applied to the target.
Replicate should apply records sequentially, thus other records will remain in the queue until these records have been processed. Since the table lacks a primary key, operations such as UPDATE or DELETE could result in a table scan and latency.
Allocating this table to a separate task is a prudent decision. It ensures other tables remain unaffected.
Regards, Desmond
Help users find answers! Do not forget to mark a solution that worked for you! If already marked, give it a thumbs up!
Besides Desmond's comment, when a batch update fails due to a NOT NULL constraint violation or other errors, Replicate has no way of knowing which of the records in the batch caused that violation. Consequently, Replicate switches to a "one-by-one" strategy of applying changes. Please be careful that "one-by-one" mode may impact the whole task performance significantly and could result in extremely high costs for a cloud data warehouse target. Check the user guide and set correct error handling policy.
Regards,
John.
Help users find answers! Do not forget to mark a solution that worked for you! If already marked, give it a thumbs up!
This is normal behavior. When Replicate encounters an issue, such as a duplicate record during the processing of a batch of records, it switches to a one-by-one mode. This mode allows changes to be applied individually, ensuring that valid records can still be applied to the target.
Replicate should apply records sequentially, thus other records will remain in the queue until these records have been processed. Since the table lacks a primary key, operations such as UPDATE or DELETE could result in a table scan and latency.
Allocating this table to a separate task is a prudent decision. It ensures other tables remain unaffected.
Regards, Desmond
Help users find answers! Do not forget to mark a solution that worked for you! If already marked, give it a thumbs up!
Besides Desmond's comment, when a batch update fails due to a NOT NULL constraint violation or other errors, Replicate has no way of knowing which of the records in the batch caused that violation. Consequently, Replicate switches to a "one-by-one" strategy of applying changes. Please be careful that "one-by-one" mode may impact the whole task performance significantly and could result in extremely high costs for a cloud data warehouse target. Check the user guide and set correct error handling policy.
Regards,
John.
Help users find answers! Do not forget to mark a solution that worked for you! If already marked, give it a thumbs up!
