How to continue replicating CDC to target after removal of source table in task and subsequently adding it back?
Hi, I have a task replicating 5 tables, sending to Event hub for downstream processing into target database.
Task setting as below:
Full load processing is ON Change processing: Apply Changes Processing is ON.
I have removed a table from the task at 3pm a day ago. Target table and data still exists in target, as at 3pm.
Now I need to add back the table to the task and resume CDC from 3pm yesterday. I do not want a reload of the re-added table to happen. And the other 4 tables should resume CDC when the task is started back.
Is below the correct way to do it?
1. Stop task 2. Select the removed table from source. Apply transformation as required. 3. From Run button, select Advanced Run Options 4. Select " Tables are already loaded. Start processing changes from 'yesterday 2:55pm'
The correct way to achieve this is to add the table back to the task, and instead of choosing "Advanced Run Options," you would set the "Start processing changes from" to "yesterday 2:55pm." This will resume CDC from the specified time for the re-added Milestone Mastercard table without triggering a full reload, while the other tables will also resume their CDC processing.
You are correct. The thing to note is that since the other tables in the task continued processing changes after you removed the one, when you start the task from timestamp after adding it back in you may get PK constraint violation errors & etc. on the tables that continued processing changes since yesterday. This error handing may slow performance until the task catches up.
Oh, I get what you’re trying to do here! So you want to add the table back to the task without triggering a full reload and without affecting the other tables that should continue with CDC. Based on your steps, here’s my take on it:
Stop the task – This makes sense to stop everything before making changes.
Select the removed table from the source – Yes, you should add the table back and apply any necessary transformations to match your setup.
Use Advanced Run Options – This is the key part. Selecting the option to start processing changes from "yesterday 2:55pm" seems like the right approach to avoid a reload. That will ensure CDC picks up from the last change without overwriting data.
Don’t forget to ensure CDC for the other 4 tables continues – When you restart the task, it should pick up where it left off for the other tables.
So yes, your approach looks good to me! Just double-check the timestamp and make sure the settings for CDC are aligned to ensure you’re resuming from the right point. It's like adjusting the flow in a system—small tweaks, but critical to keeping everything running smoothly without any hiccups!
The correct way to achieve this is to add the table back to the task, and instead of choosing "Advanced Run Options," you would set the "Start processing changes from" to "yesterday 2:55pm." This will resume CDC from the specified time for the re-added Milestone Mastercard table without triggering a full reload, while the other tables will also resume their CDC processing.
You are correct. The thing to note is that since the other tables in the task continued processing changes after you removed the one, when you start the task from timestamp after adding it back in you may get PK constraint violation errors & etc. on the tables that continued processing changes since yesterday. This error handing may slow performance until the task catches up.
Oh, I get what you’re trying to do here! So you want to add the table back to the task without triggering a full reload and without affecting the other tables that should continue with CDC. Based on your steps, here’s my take on it:
Stop the task – This makes sense to stop everything before making changes.
Select the removed table from the source – Yes, you should add the table back and apply any necessary transformations to match your setup.
Use Advanced Run Options – This is the key part. Selecting the option to start processing changes from "yesterday 2:55pm" seems like the right approach to avoid a reload. That will ensure CDC picks up from the last change without overwriting data.
Don’t forget to ensure CDC for the other 4 tables continues – When you restart the task, it should pick up where it left off for the other tables.
So yes, your approach looks good to me! Just double-check the timestamp and make sure the settings for CDC are aligned to ensure you’re resuming from the right point. It's like adjusting the flow in a system—small tweaks, but critical to keeping everything running smoothly without any hiccups!
