Databricks Lakehouse (Delta) Target connection : Does the qlik job permit both full load and CDC (apply changes) on the same delta object?

[Rak]

I am referring to https://help.qlik.com/en-US/replicate/November2023/Content/Replicate/Main/Databricks%20Lakehouse%20(...

Will the creation of a Qlik replication task handle both full load and CDC (apply changes) on the same object when the data is moved from a SQL server to DataBricks Delta Lake housed on GCP?

Scenario is :

Qlik task to replicate data from Sql Server to Databricks delta

1. table1 --> Full load - 100 rows loaded to databricks delta (table1)

2. cdc on table1 --> 10 inserted , 10 updated -Would the databricks delta table have 110 rows? Does the same qlik replicate task perform the merge as part of apply changes?

I would like to know if it is possible for the same Qlik replicate task to apply modifications to the delta target and complete the full load in one go. Does combining the data from the landing to the target tables require Qlik Compose?

[john_wang]

Hello @Rak ,

Thanks for reaching out to Qlik Community!

While working with Databricks Lakehouse (Delta) as a target endpoint, the Apply Change replication mode is supported, we need not Compose for the CDC data merge. This is different than the  Databricks (Cloud Storage) endpoint.

However please take note that The Apply Changes replication mode supports tables with a Primary Key/Unique Index only. Tables without Primary Keys/Unique Indexes will be suspended at the Change Processing stage. Or you can set the task to Store Changes replicate mode and utilize Compose for the change data processing.

Hope this helps.

John.