Re: How to Handel Large data set while loading in qlikview.
One important note - if loading from QVD takes that long compared to loading from DB, something is not right with the world. Reading from the properly-prepared QVD should be significantly quicker than you describe, even with a large number of rows, assuming your server has enough RAM to handle the data. If it is going as slowly as you describe, there are two options:
A. You are not correctly using the QVD process:
1) Load your data from the database.
2) Make any modifications to the data necessary. In particular, any elimination of data (WHERE conditions) or grouping (GROUP BY) should happen here.
3) Save the fully-prepared data to QVD.
4) When reading the QVD, do so without adding any new filtering/grouping conditions.
Note that it may be better for you to use incremental reload techniques, save each new increment of data into a new QVD file, and when loading, use a script to loop through all these files and read them one at a time.
B. Your machine does not have enough RAM to handle the data volume, and the additional time is being used for swap-writing to your hard drive. You should easily be able to detect this by watching the memory usage during loading - if it's constantly at or close to 100%, this is likely your problem. If this is the case, try adding RAM. Switching to faster (SSD) disks might also help to some extent, and will also reduce some of the time required to save your document.