QVD taking over an hour to store

[#1]

I have a QVW that stores data to a QVD.  Storing the data takes over an hour.  I pasted the end of the log from when this QVD is created.  The QVD is large - 18,844,497 KB when it ran today.  Does it seem normal that it would take this long for a QVD to store? 

9/19/2011 11:06:53.1132155 Information 9/19/2011 11:06:52 AM: 2030        if(num(WeekDay(Today())) < 5) then

9/19/2011 12:15:45.7115274 Information 9/19/2011 11:06:53 AM: 2031          store workListRecs into E:\QVD\Historical\WorkListRecs_Hist.qvd

9/19/2011 12:24:22.2875726 Information 9/19/2011 12:15:45 PM: 2034          DROP table workListRecs

9/19/2011 12:24:22.3967754 Information 9/19/2011 12:24:22 PM:      Execution finished.

[#2]

What is the size of the QVD?

[#3]

The QVD is large - 18,844,497 KB when it ran today.

[danielrozental]

you might want to consider splitting QVD files by date, month, year or whatever other field that makes sense so you would update and store just that QVD and not all the info.

[#5]

I guess it is normal to take an hour to process this QVD.Make sure that you don't have any parallel process running in server while you process it. It might be help to process it faster.  Our dev server takes 1 hour to reload a takes while prod can do it with in 30 min.So the hardware and available resources also matter in this scenario.

[Oleg_Troyansky · Accepted solution]

The file is truly huge, so it might actually needs that long to save... Some of the other things to look into:

- is the file being stored on a remote machine over a network?

- What's the network speed?

- Is is a SAN storage?

- If the hard disk is local to the machine, can a faster hard drive be installed?

with such a large data volume, every bit of performance matters...

It might be advantageous to slice the file into several files. It might be faster to save 10 smaller files than one huge file.

Another consideration could be re-structuring your incremental load. Traditonally, during incremental load we combine "old" data with "new" data into a single table, which evendually becomes "old" data for the next load. In your case, you might decide to keep "old" data where it was and store every incremental piece separately. You might see a huge improvement of performance...

take care,

Oleg Troyansky

[#7]

Thank you for the help everyone.  I ended up splitting the files, and this helped a lot.