[Data Pipeline Projects] Recovery Procedures When a Truncate Occurs in the Source and How to add a source table to an existing pipeline.
I'm using Data Pipeline Projects to replicate data from Oracle to Snowflake via S3.
- There is 1 landing task and downstream 1 storage task.
- The “Historical Data Store (Type 2)” option in the storage task is turned OFF
Since loading data into Snowflake via S3 is a mandatory requirement for the customer, we doesn't use Data Replication Projects.
I couldn’t find a solution in the Help documentation, so please advise how to operate on the following scenarios during operation.
1. When a table is truncated in the source Oracle database , I want to restore the Snowflake "__current" table to match the source data.
In a Replication Project , truncation is synchronized , so no action is required.
However, in a Data Pipeline Project, truncation is ignored in landing task.
2. I want to add a new table to the existing landing task and storage task.
In a Replication Project, if I perform the following steps , the table is fullload automatically and is replicated to target.
・Stop the task
・Add “Select source data”
・Restart the task
However, in a Data Pipeline Project, if I perform the same steps in the Landing task , the table is not landing automatically , nor the table is not added in the Storage task.
