Compose DW ETL Tasks

[Al_gar]

For a DW in a Compose project:
1. we have around 21 ETL Tasks mappings created by a previous consultant, but I notice that the default pair (i.e. Landing_source and Landing_source_CDC) that I assumed where generated by Compose have no mappings selected.  Can you create as many mapping on a DW as you want and not use the "default" ones? What are best practices regarding DW ETL Task mappings?
2. For the "_CDC" ETL task, I watched some Qlik videos where the "_CDC" and DataMart tasks were added to a workflow. Can this workflow be trigger by that incoming changes? or, does always need to have a schedule in order to process any changes?
3. AFAIK, "_CDC" task process changes from Replicate's _ct tables. How does it process changes in the Landing zone that were not created with Replicate?
4. Is there any example for building a DW using tables w/o foreign keys or the auto discovery future?

[Nanda_Ravindra · Accepted solution]

@Al_gar 

For #1,

-By default Compose enables it based on the task type( FL vs CDC) For FL the mapping gets enabled and for CDC both duplicate and mapping gets selected, "Handle Duplicates" checkbox tells Compose that there could be duplicate records for each key with different FROM DATES (FD)

-You can create many mapping on a DW  and not use the "default". But if it's a complex mapping then I would suggest you create a new task for your complex mapping and make sure to map all the columns that you added.

- We recommend using our default mapping and adding your additional mapping on top of our default mapping, but as I mentioned above I would suggest creating a new task if your mapping involved complex logic

For#2: Workflow can only be run by schedule or by manually executing it - workflow can not be triggered by incoming changes.

For#3: _CT tables are always associated with replicate, and Compose runs the CDC based on this _ct table.

For #4: MostCompose customer uses Replicate to bring source data to Compose. But we also have the option to import it from the Erwin model if you don't want to do it with auto-discovery - see blow.

Hope this helps!

Thanks,

Nanda

[Al_gar]

Thanks @Nanda_Ravindra. For some Compose ETL task mappings we use SAP HANA with Log based CDC and I see that there's a limitation in section "5.1  Defining a Qlik Replicate task" : "As Compose requires a full-after image to be able to perform Change Processing, the following Replicate source endpoints are not directly supported (as they do not provide a full-after image): SAP HANA (log based)". I suppose that  Compose will not rely or process any data from the _CT tables, right? If so, do you know if there's a workaround the other customer may have used in this scenario?

[Nanda_Ravindra · Accepted solution]

@Al_gar you can use the SAP HANA (trigger-based) endpoint in replicate to get around this scenario but nothing from Compose. Hope this helps!

Thanks,

Nanda

[Al_gar]

Do you know if there's any plan to support full-after image for SAP HANA CDC Log-based endpoint in upcoming releases?

[asuka55]

A couple days ago, I came across this post which suggested that using Pandas was quite an inefficient way to build out an ETL pipeline.

As someone that’s just recently transitioned into DE it’s the only way I know how. What are the better alternatives out there and what would be a good starting point for me?