I have a task that perform a full load once a month. In the past, I have experienced multiple instances where the EC2 instance hosting replicate task was restarted in the middle of the task execution as part of the scheduled monthly maintenance.
When this happens, once the EC2 instance comes back online, the replicate task automatically restart from the beginning. As a result, duplicate records are inserted into the target tables for the data that had already been loaded before the interruption.
To prevent this, I configured abnormal_count": 0 under recoverable_error_behavior and tested the scenario by simulating and unexpected restart. However, even with abnormal_count set to 0, the task still restart automatically after the EC2 instance is back online.
Is there a way to prevent the task from automatically restarting after the EC2 reboot, or is there a recommended approach to handle this scenario and avoid duplicate data? Any suggestion would be greatly appreciated.
Restarting the EC2 instance during task execution causes Replicate to start the task in Fresh Start mode.
Full Load + CDC tasks restart the full load instead of resuming from the last checkpoint.This happened because it error occurred as there is a error status code mentioned below it the recoverable_error_behavior.abnormal_count=0 setting does not apply because a service restart is not treated as a recoverable error.
Recoverable error settings only handle runtime errors (for example, connection interruptions), not Replicate service or host restarts.
You may choose below steps to avoid Future occurrence
Manually Stop all running Replicate tasks.
This ensures the current task state is saved correctly.
do the maintenance
Start the Replicate service.
Resume the tasks to continue from the last saved checkpoint, or perform a Reload only if required.
