Full Load Vs CDC ETL Set

[ubanerjee]

What makes a ETL Set Full vs CDC? The only difference I see in the UI is the suffix _CDC. If I create a new ETL Set and end the name with _CDC would it become a CDC ETL Set?

[Brian_Jones]

There are differences in the ETL code that Compose generates for each. At a high level, the CDC ETL Set differs from the Full load in the following ways:

The CDC ETL set is built to process changes from the __ct tables that Replicate uses to deliver changes to the Compose landing zone when the Replicate "Store Changes" option is selected. Even though you have mapped to a table called Customers, the CDC ETL set assumes Replicate has put a Customers__ct table in the landing zone; whereas the Full load ETL set reads from the Customers table in the Compose landing zone.

The CDC ETL set is assumed to be processing changes, so the duplicate processing that is optional in the Full Load ETL set is automatically applied. 

[TimGarrod · Accepted solution]

To add to Brians comments - no, you cannot just append _CDC to make the etl set generate code for CDC processing.  Unfortunately in the UI (other than being called CDC) there is no marker / config to create a CDC ETL set or to signify it is one other than the out of the box naming convention.  

If you wish to had multiple CDC ETL sets, I recommend un-assigning mappings to an existing CDC ETL SET and then selecting Duplicate.  (Note you can do this for a full etl set also and then create a CDC ETL Set from there). 

[ubanerjee]

Thanks Tim and Brian,

I was trying to create a new ETL set and configure it as CDC.