2 Replies Latest reply: Feb 21, 2017 9:01 PM by Vladimir Komarov RSS

    How much data size could QVD handle?

    Haikuo Yu

      How much data size could QVD handle?

      Is there any data size limitation?

      I have a data set with 200 million rows and 50 columns. Is it too big for Qlik QVD? Is it too big for Qlik sense to load?

      Thanks!

        • Re: How much data size could QVD handle?
          Vladimir Komarov

          There is no limitation for data to be stored in QVD(s). The only possible limitation is your PC's system resources  (HD size and Memory size).

           

          There were some reports that QS saves large QVD files unproportionally slower than the smaller ones, but I believe it's also related to HD's condition (fragmentation, etc) and available HD space....

           

          Regards,

          Vladimir

          • Re: How much data size could QVD handle?
            Rob Wunderlich

            A QVD can handle that amount of data no problem,

             

            Although a single QVD can handle all that data there are situations where is advantageous to break the table into multiple QVDs, segmented by a period or dimensional value, such as a Month or Region. Reasons for doing so include:

             

            1. When QVDs need to be updated by incremental load. It is usually faster to update a few smaller QVD files than a single large file.

             

            2. When the QVD consumer may want a limited time period or segment of data. If you only want the last 12 months -- it is faster to load the specific 12 monthly QVDs than loading and filtering a single QVD that holds 5 years.

             

            3. Rolling off old data. If you want to keep a rolling 5 years, it can be faster to delete old monthly files instead of trimming data from a single file.

             

            The Month case used above is only an example. You can of course use any segmentation that makes sense. If your reload times are acceptable with a single QVD, by all means stick with a single QVD as it is easier to script with.

             

            -Rob