Incremental load for single big table

[bharath28]

Hi All,

We follow the 2 tier architecture. The qvd generator(qvw) creates many qvds and reload time depends on the date range.

There is one qvd that contain fact table data which takes around 80% of the total time of qvd generator(qvw). The fact table contain around 60 Million rows for 3 years date range.

-Since the fact table takes maximum time, I want to create this qvd using sepearte qvd generator(qvw), so that if build fails, other qvds wont effect.

Also, I thought of creating multiple fact table qvds, each qvd containing set of rows, so that i concatenate those qvds in final dashboard. Do you think it is the right approach??

I never implemented incremntal loading, so i dont know whether i can use it to build my fact table qvd. Do you suggest implementing it in my scenario??

[kaushiknsolanki]

Hi,

I would suggest that you should go for a new Extractor which will create this QVD and use the incremental load so that your reload time will also decrease.

For incremental loading there is lots of stuff available on this community, just do a search and I am sure you will get what you want.

Regards,

Kaushik Solanki

[Peter_Cammaert]

An Incremental load is the best solution to reduce your load delay. You'll have to figure out how you can detect which records are new/changed and just load/add those. But the simplest approach is to use a record timestamp in your database table. An incremental load will first lift the last-reload-time from the QVD that is present on the QlikServer machine, and then only load records that are newer. Then the historical qvd will be added to these records and the result will be saved as the new qvd.

I would not split the resulting qvd, as this may make the reload script incredibly complex and might take longer to use/reload these partial tables.

Whether you should take the facts reload out of the other reload chain depends on your situation. It sounds like a good idea while you are converting the reload of the large facts table. And you could run some reloads in parallel that way (also a possible time-saver)

Peter

[mohammadkhatimi]

Hie...

PFA.

Hope this will helps you...!!

Regards,

Mohammad

[bharath28]

Thanks for all the suggestions!

Sorry, I just realized that the fact table data varies based on the client for whom we want to create the qv document.

so, if i want to implement incremental load, i need to keep ALL previously created client qvds in a folder, so that qlikview will pick the particular client qvd and check its records and load only the new or updated records from source database?

eg-

qvd folder-->

         facttableX.qvd

         facttableY.qvd

         facttableZ.qvd

qlikview script-->

SET vClientID = 'X';  //changed from UI

FactTable:

load *

from facttable$(vClientID).qvd

Also, we sometime create the qvd for 1 year or 2 years or some date range, for specific client.

Eg-

So, if my old qvd has 3 years data and i want to create new qvd for 2 years data and i expect some changes in source database, Does incremental load support this?