In Compose it would be great if there was an option to specify to not archive change tracking tables from the Replicate landing area and just leave them where they are. Currently there are only options to either delete or archive in the project connection settings. In Snowflake since the data is automatically partitioned, archiving data will not improve performance and simply adds additional overhead which is not required. You could add an option in compose to only process data since the last data warehouse run or reprocess all data. The proper clustering key could be added in replicate when Snowflake is the target to ensure partition pruning is leveraged in compose when reading the data and only the data needed is read instead of having to search through the entire data set. This would keep performance consistent without having to perform the additional steps to move and delete the data . We have encountered several cases where we needed to reprocess the data and had to manually take the data from the ARCHIVE schema and put them back into the change tracking tables to ensure all data was reprocessed. Having this option would also allow people to be able to use the same landing area for different projects for various dev and learning purposes and avoid having to create multiple copies of the same data.
