QVD Necessary

[help4qv123]

Hi Friends,

I have got a small discussion, as we all know we use QVD for faster reload and incremental load . my question is after completion of development when i publish my dashboard in server there will be 2 jobs..one will be loaded from my extractor(source is Database) and QVD's are created and the 2nd will be from my dashboard (source is QVD).

So we say that to reduce reload time or for faster reload we use QVD's. But 1ce the dashboard is loaded the data is loaded in QVW.

so for what we use QVD.

Now, there are 2 reloads (Jobs):

1. Extractor ( Source is Database) ( Reload time is 25min)

2. Dashboard (Source is QVD) ( Reload time is 2min)

So instead of two reloads , y cant i make it one. where my connection will be direct from database???????Becze anyhow i have to refresh the QVD , for that i have the hit the database everyday......

[tresesco]

If you don't have to consider incremental reload (possibly if you have kind of static data) and any major transformation of data in qv, you could perhaps skip using qvd. But there could be cases (and that is the case with most of the real life scenario) when you would need Incremental Reload and some Transformation of raw data before it is produced before the end users.

Case-Incremental Reload : If your organization has 100M rows of histirical data and 100 rows of additional daily data, you might not want to read the entire data everytime from the source because that would be bad in terms of performance. Rather you could save 100M+100+...data incrementally in qvd and read only 100 new daily records from source. This will increase the performance.

Case-Transformation: The data from source is not always presentable to the users as-it-is. So you need to transform it. You can fetch the data from source everytime you need to refer from source but that again would not be good performance wise. Better, reaad it once from source and store it in a temp qvd and refer it as many times it requires. It would be faster. Moreover, reading the finally transformed qvd directly (called optimized load) is very fast.

Hope that makes sense.

[avinashelite]

Connecting to DB everyday is a valid point. But when you consider it as an operation, we can use the QVD creation file as the business layer where we can implement all our complex logic related to data and transofrmations.

you can handle the QVD creation and dashboard refresh at same time , it all depends on the logic how you implement and performance .

[help4qv123]

So with out QVD incremental load is not possible. right?

if there is large amount of data we prefer incremental load with QVD else not necessary?

[santhoo_san]

When you do direct reload from the source, it interrupts the database while pulling data for QlikView. That in turn effects slow down of database for other users of the database.

[buzzy996]

So with out QVD incremental load is not possible. right? --- it's not that mean,u can do with out incremental load but tht's take's to complete ur load 27 minutes for very load,tht will create a burden on ur source system and network traffic.

if we use incremental load tht will reduce burden on source system/network trafic because no need to full complete data every data,only u can fetch updated records/latest records.

if there is large amount of data we prefer incremental load with QVD else not necessary?--- this incremental load will helps,but again tht's depends on wht data u want to see in .qvws.

[Peter_Cammaert]

IMHO there are two issues that - in your case - shouldn't be related: (1) performing incrmental loads using QVD's, and (2) a two step document update. If I get it right, you're asking yourself whether you really should use two stages to update a single document instead of a single stage. The answer depends on whether this is the only document that uses data from your database.

You can implement a perfect incremental load, using QVD's in the same document that will be published to your end-users. That would be a single stage (single script) update: your dashboard will access DB and optionally some QVD's, figure out what it needs and in what form, and update the internal data model.

Now imagine you have to design a second document that will use a lot of the same tables as the first one. An important question arises: will you do the same incremental LOADs again for this second document (thereby accessing your DB again in the same way for almost the same data and probably immediately after the first access)? Or will you implement a data layer that takes care of the incremental loads and presents the result as a consistent data set to both dashboard documents (through QVD's)?

In my experience, it's a better move to implement a multi layer approach from the start than to try to apply it on an unwilling existing QlikView environment when the need arises.

There may be other issues that come up when facing these situations (load times, intra day updates, consistency accross QVD's and QVW's, etc.) but they don't seem to matter at this point in time.

Best,

Peter

[jagan]

Hi,

Without QVD it is not possible to implement incremental loading, we need to store somewhere the previously loaded data.  So QVD is the best method to store the previously loaded data.

If you are using the same data for example Customer Master, then we will store this in QVD because this data is used in almost all the Dashboards.  Instead of going to database everytime we will fetch from QVD

For example if we have 10 Dashboards and we are using Customer Master in all the dashboard then first time we will pull from Database and store in Database and after that in the remaining Dashboards the customer info is read from the QVD.  This way we saved 9 hits to the database.

Always try to reduce load on database, because databases are used for transactional purposes, it affects the business.

Hope this helps you.

Regards,

Jagan.

[help4qv123]

My Qus was with out QVD's can we do incremental load...

[help4qv123]

Thankx Peter,

Can u elaborate more on multi layer approach