5 Replies Latest reply: Jul 4, 2013 2:39 AM by shivaji ch RSS

    Merging QVD Files

      Hi Everyone ,

       

      I was having 15 million records in one table , when I tried to load and store them in qvd my database ran out of memory , So as a solution I have loaded data in chunks of 2.5 million and stored them in 6 qvd files . Now when i try to load these files and merge them into single qvd my system's physical memory which is 8GB gets fully consumed (System got stuck) , What are the possible ways to merge these files , and later how can I effectively use that single file(biggest one). Here is the code through which I am merging those qvds

       

      [Incremental]:

      NoConcatenate

      Load *;

      Load * from

      '..\Data Files\File1.qvd'(qvd);

       

      Concatenate

      Load * from

      '..\Data Files\File2.qvd'(qvd);

       

      Concatenate

      Load * from

      '..\Data Files\File3.qvd'(qvd);

       

      Concatenate

      Load * from

      '..\Data Files\File4.qvd'(qvd);

       

      Concatenate

      Load * from

      '..\Data Files\File5.qvd'(qvd);

       

      Concatenate

      Load * from

      '..\Data Files\File6.qvd'(qvd);

       

      STORE [Incremental] into '..\Data Files\File.qvd'(qvd);

        • Re: Merging QVD Files
          Kaushik Solanki

          Hi,

           

               It doesnt matter whether you are using single qvd of 10gb or 5 Qvd of 2 gb each.

           

               When ever you load the data from qvd, all data will be moved to RAM and then other calculations will be done if any.

           

               So the only option i see is you increase the configuration of your computer.

           

          Regards,

          Kaushik Solanki

          • Re: Merging QVD Files
            Amay Patil

            Hi,

             

            I would suggest to load only the required fields that are useful into QVD's.

            If you are doing it already, then follow kaushik's suggestion .

             

            Regards,

            Amay

            • Re: Merging QVD Files
              shivaji ch

              Hi


                             Firstly these 6 QVDs should have some common ground on which to use them as primary keys.

              Considering this, you will joins has transformed everything into a single data source.

               

              example:

               

              Concatenate:

              Load * from

              '..\Data Files\File1.qvd'(qvd);

               

              Left Join (Concatenate)

              Load * from

              '..\Data Files\File2.qvd'(qvd);

               

              Left Join (Concatenate)

              Load * from

              '..\Data Files\File3.qvd'(qvd);

               

              leftJoin(Concatenate)

              Load * from

              '..\Data Files\File4.qvd'(qvd);

               

              left Join(Concatenate)

              Load * from

              '..\Data Files\File5.qvd'(qvd);

               

              Leftjoin(Concatenate)

              Load * from

              '..\Data Files\File6.qvd'(qvd);

               

              Store Into Concatenate [..\Data Files\QVDTransformed.qvd];

               

               

              Regards

              shivaji