Need understanding on Landing, Storage blocks and different tables that are created in the target.

[akaradhya]

**New to Qlik cloud data integration**

I have successfully set up the QCDI on my tenant. In the task I've setup my source is MySQL and my target is Snowflake. 

1. When I set the source as MySQL and my platform as Snowflake, it automatically creates Landing and Storage blocks. I want to get an understanding on is it mandatory to have both the blocks and what is the significance of it. As far as my understanding goes, I can still specify all the required transformations in the Landing block and store the data in the Landing block itself rather than having a Storage block. Also, pl share if there is specific use case for creating the landing and storage blocks. 

2. Post running the storage task, I see in the target that there a multiple tables that are created...like tablename, tablename_live (I am guessing this is created because I've selected Live views in the settings), tablename_changes. My question is, are these tables being created from the data in the Landing zone. I checked the SQL on these table and I can see it is referring to the storage_internal table. If so, is the storage_internal table picking the metadata from the Landing zone. 

Also, in the storage section the data is present as views. Is that the only option or can I also create it as a table in the storage area. 

3. Also when I have the history enabled in the task, there are additional tables that are being created in Snowflake. Request you to share some documentation if available on what basis are these tables being created. 

Thanks and sorry for the long question. 

[TimGarrod]

Hi,   this is a great question :).

For all three , I recommended reviewing review the documentation that describes the architecture, schemas, tables and views created, schemas here - https://help.qlik.com/en-US/cloud-services/Subsystems/Hub/Content/Sense_Hub/DataIntegration/Storage/... 

Some additional color - When delivering real-time change data capture (CDC) from databases using the Data Movement Gateway, Landing is where changes are inserted into the cloud dw target platform and staged for processing by the Storage task.    Storage is your long-term persisted storage which can optionally provide historical data (akin to a type 2 of data changes captured over time) as well as the 'current view' or updated data. 

Live views provide for a 'delayed merge' architecture where you can get the benefits of real-time data without having to run merge as frequently against your cloud data platform to help reduce costs. 

Additionally landing provides a method where a full load can be performed and a comparison to the target (storage) ensures history can still be tracked, if you don't have the means to support log-based CDC, thereby reducing any data down-time for downstream pipeline components or consuming applications / processes.

Note that if you are only running full load, and don't care about historical storage of your data, or further transformations, it is entirely possible to just create a landing process.  However, once your requirements include CDC (with incremental updates in the target),  long term historical storage to support type 2 history requirements or transformations and data mart creation - the storage task is an integral component of the data architecture.

Hope this helps!