Skip to main content
Announcements
Have questions about Qlik Connect? Join us live on April 10th, at 11 AM ET: SIGN UP NOW
cancel
Showing results for 
Search instead for 
Did you mean: 
alexandernatale
Creator II
Creator II

ETL, QVD...a little scheme

Good morning everyone,

I have a doubt that has haunted me for a few days.


 I am a QlikSense SaaS user. I have always wondered why I use qvd files.

Until now I have always created apps according to this logic:
- I extract data from sources (ex: github)
- through the script editor I create the data model.

In this way I bypassed the creation and storage of qvd files.
I am in constant training and therefore I have "discovered" the qvd files.

I have read on this same forum the advantages that derive from it and I would like to approach the subsequent apps (testing and training) using this paradigm.


However, I am confused about its practical use and in particular how it fits into the ETL process.


I have created a diagram that I report in the figure to verify my - I hope correct - interpretation of a data stream from multiple sources.

ETL.jpg

 

Phase 1: Extraction (E)

In this phase, trivially, I extract the data that I have to collect from the various data sources to store them in a data warehouse.

Phase 2: Transformation 1 (T1)

In the storage phase within the data warehouse I can perform some transformations (format fields, division of columns, etc.). Typically, through queries in mysql.

Phase 3: Transformation 2 (T2)

From the data warehouse, through the QS SaaS connections, I read the data and transform it further (data format, creation of flags, division of tables into multiple tables, etc.). In this phase I create the qvds: one for each table of the data model that I want to create (this, I assume, to create and then import an optimized qvd)

Step 4: Loading (L)

In this phase I load the various QVDs generated in order to operationally implement the data model that will then be the basis of the subsequent reporting and dashboard creation phase

Is this process just described formally correct? The steps and usage of qvd file is correct?

I look forward to your answers!

Labels (4)
1 Reply
hic
Former Employee
Former Employee

QVD files are NOT necessary, and should in some cases be avoided.

However, there are some advantages with QVD files: First, they can be loaded VERY fast. So if your nightly window for refreshing the apps is short, it would be useful to prepare the data in the QVD format in a separate process. Also, for incremental loads, the speed is important, and QVDs are useful.

Secondly, if you want to create an intermediate storage of prepared data, the QVD file format is a natural choice. But you could just as well use a data warehouse. However, a data warehouse is a "heavier" solution that involves more work and more preparations.

In your case, you could in principle to all your transformations in the data warehouse, and skip the QVD step.

Good Luck!