Qlik Compose Data Warehouse - Incremental Load Patterns for Non-Replicate Data Sources

[TimGarrod]

Qlik’s Data Integration Platform provides functionality to support near real-time data delivery from traditional sources, files, and some SaaS environments via Qlik Replicate. Qlik Compose for Data Warehouses is integrated with Qlik Replicate to automate the incremental load process of the data warehouse based on change data capture (CDC) from the source system.

However, there are certain scenarios where the automated incremental load processes cannot be leveraged. (For example, SAP Hana Log based replication, third party data ingestion or cloud data sharing programs).   

In those instances there are extract transform and load (ETL) patterns that can be implemented in Qlik Compose to support incremental data loads. This paper will describe these scenarios and show how to implement custom incremental load patterns in Qlik Compose and how those patterns can be applied to query or view based mappings.

[Al_gar]

I'm following the incremental filter load pattern approach from this whitepaper. We have many views mostly from SAP HANA tables. Can you please provide answer for the following questions/assumptions:

1. On p.12 states to apply a filter in the mapping's "Build Expression" window. Should this step be omitted if the filters are added to the query/view as shown on the query of p. 16?
2. On p.12, section "Extending the Configuration to Process Multiple Sources" when managing separate sources ChangeSeq independently, does this also refer to the scenario where 2 or more mappings are under the same logical entity? or does it refers only when there are multiple sources in a single ETL mapping? (e.g. HANA, SQL, Oracle, etc.)
3. On p. 16, do we need keep track of the ChangeSeq for all the tables joined in a query even when it's not a driving table?
4. Is there any approach to keep filter changes when source query/view is from full load (truncate and reload) table?

Qlik Compose 2023.11.0.69; Qlik Replicate 2023.11.0.159; Target SQL Server 2019

Thanks.

[Sonja_Bauernfeind]

Hello @Al_gar 

I recommend posting your query directly in our Qlik Replicate forum to make use of the wider reach into our community and our active support engineers. When posting, include the symptoms of your issue, any error messages that you have seen, and what troubleshooting steps you've already taken. Feel free to reference back to this article as an example of what you tried.

We also invite you to make use of our chat function if you require more direct assistance or a case to be logged.

All the best,
Sonja