This request is to have finer controls for re-setting / reloading data for a specific table. The new integration with Rep for a full / re- load is great, but in general selectively processing a table is an issue in ComposeDL today and in gen2.
For example - if I have the a scenario where I have to alter a single tables definition (lets say I add a calculated column or add a column to the PK and I have the initial full load and history available in the CT ).
I need to reload data from the full load / CT data for 1 table, I would have to create a new / special FULL LOAD process, special CDC process and then sink up the 2 CDC tasks. This is a pain in a real world production environment as you typically can't just go into prod and start creating new tasks.
Currently Compose pulls the MAX(partition) from the cmps_status table for the entire task. Instead of processing based on the task, we should process based on the individual TABLE. This would allow a process where I could reset the partition names in the cmps_status table to force a reload of a single table WITHOUT having to perform a full load in Replicate.
Akin to what happens in the "Collect entities with LOADED data" - but perform that same logic with "collect entities with changed data".
Along with this - it would be great if the CDC load could detect FULL LOADS for NEW TABLES in the ETL task and not just those that have already been processed.
This would go along way to improve operational running of Compose - adding new tables through deployment and controling table loads if needed. (One big complaint from customers who dropped ComposeDL was that operational / deployment etc of C4DL was painful).
