topic Breakdown the reload script into Data Sets and Chunks to achieve independent reload cycle in QlikView

Breakdown the reload script into Data Sets and Chunks to achieve independent reload cycle

Mon, 20 Dec 2010 07:45:42 GMT

This is what of my collegue wants me to achieve it. Could you please tell me how it can be achieved. I am not able to understand what exactly he meant by the following lines below? Hope some one can come up with what exactly he wants and also let me know how it can be achieved.

" Breakdown the reload script into Data Sets as the reload script is getting quite large, I would like to break it into chunks so that we can achieve independent reload cycles"

Thanks in advance!

Breakdown the reload script into Data Sets and Chunks to achieve independent reload cycle

Mon, 20 Dec 2010 10:11:21 GMT

" Breakdown the reload script into Data Sets as the reload script is getting quite large, I would like to break it into chunks so that we can achieve independent reload cycles"

It means the reload script is pulling too much data in single memory table and thus the memory table is very large.

Hence load data in small pieces and store in multiple QVDs

ex if you are pulling for year 2010 then inspite of loading full data for 2010 load for 12 months one by one and store 1 month and drop the memory table for that month then load next month.

At last concatenate all 12 QVDs in final application.

Breakdown the reload script into Data Sets and Chunks to achieve independent reload cycle

Mon, 20 Dec 2010 11:38:37 GMT

Its a very good input from subhasheeh. This type of approach is very usefull when we have a very large data and it takes long tine to reload.

1. You should make qvd at different level /Phase.

2. Now All of this QVD can be Called in Final application. - Optimized QVD takes very fewer time to get loaded as compare to raw qvd.

Breakdown the reload script into Data Sets and Chunks to achieve independent reload cycle

Thu, 23 Dec 2010 10:30:24 GMT

Hi Shubhasheesh Chakraverty

Yes I totally agree with your point. But even to create the qvd we have to reload the data. What if we have created the QVD later there has been some changes in the data later on. In this case we may to need to reload all the data again and create QVD.

There should be some way to identify where the changes are there or else we need to reload all the data again. It may not make any sense.

There should be some way for identifying the changes also. Am I right?

Breakdown the reload script into Data Sets and Chunks to achieve independent reload cycle

Thu, 23 Dec 2010 10:33:44 GMT

Thanks Ashwin

I totally agree with your point. But I feel will make sense only if we can get to know where the changes are made and loading only the new data in to the QVD instead of loading all the data again and again.

Breakdown the reload script into Data Sets and Chunks to achieve independent reload cycle

Thu, 23 Dec 2010 11:03:43 GMT

Hi,

If you want that the data already loaded in qvd should not be changed by full reload then

There is a function Reloadtime().

If your fact table has got any field for date then use a where clause like where (datefield > Reloadtime())

If you want to reload only changed records then you should go for partial reload using add / update keywords. Reference manual has nice examples for it.

Regards,

Shubhasheesh Chakraverty