Why Does QCDI Create Multiple Views During Onboarding Storage ?

[harsh2]

Hello Team,

While using Qlik Cloud Data Integration (QCDI) for warehousing, particularly the data pipeline feature, we observed that multiple views are created during the onboarding storage stage. We understand data structure of each object by reviewing them; such as maintaining history and other related aspects. However, we seek clarification on why QCDI creates multiple views, especially since this does not seem to be a feature that can be disabled.

Scenario:
We loaded a test table into Snowflake and ran the data pipeline using the default settings. On the target side, multiple objects were created, as seen below snippet

Within the onboarding_storage and onboarding_internal_storage schemas (created by the onboarding task), there are a total of 10 copies of datasets: 7 views and 3 tables.

It would be helpful to understand the rationale behind creating these views and tables and their intended purpose.

Attachment:

Please find the attached snippet illustrating the observed objects.

Thanks & regards,
Harsh Patel

[sureshkumar]

Hello @harsh2 

Views concept in QCDI as same as views concept in Qlik Compose (Data Lake)

Kindly refer the below article on views

Working with views | Qlik Compose Help

Hope it helps.

Regards,

Suresh

[harsh2]

Hello @sureshkumar ,

Thank you for responding to the post.

As mentioned in the post, we have referred to that document. However, what if we don’t require these copies of datasets?

For instance, if the customer has 300 tables, this would result in 3,000 copies of those datasets. Is there any documentation available to clarify why these are being created in the warehouse?

Thank You,

Harsh Patel

[sureshkumar]

Hello @harsh2 

There is no document on this, please open a support ticket and we will check with our internal team on this.

Regards,

Suresh

[AntoineRichard]

Hello @harsh2, I understand that your concerns is about "copies" of the data. Allow me to explain:

• Landing: The 2 tables that we create are not copies. The "test" table stores the data that was loaded initially. Then, any change that happens to your source data is replicated in the "test_ct" table ("change tracking").

• Storage: The storage contains different tables that store the current version of the data and the prior versions of the data. This is not a copy but different values of the data over time. Note that the internal schema is not supposed to be accessed by users. 

• Finally, views are purely virtual, they do not store any data. The views in the "onboarding storage" schema are the consumption layer. They hide the internals from the data consumer.

In short, there's only two "copies" of the data (with a different structure): one in landing, one in storage.
The landing layer exists for replication performance reasons.

Please refer to this documentation (and the video on "live views and delayed merge") for more details: https://help.qlik.com/en-US/cloud-services/Subsystems/Hub/Content/Sense_Hub/DataIntegration/Storage/...

[TimGarrod]

In addition to the info Antoine provided, please note if you do not wish to have some of these features you can disable them in the task settings. 

For example if you do not require the use of Live Views or a Type 2 of your data, you can disable those capabilities in the Storage task settings. (Settings > General and Settings>Views)

The task settings will set the default behavior for all datasets in the task.  Please note, if you wish to override the default you can also do this on a dataset by dataset basis. 
Dataset-> (3 dot menu)->Settings