Merging QVD Files

[#1]

Hi Everyone ,

I was having 15 million records in one table , when I tried to load and store them in qvd my database ran out of memory , So as a solution I have loaded data in chunks of 2.5 million and stored them in 6 qvd files . Now when i try to load these files and merge them into single qvd my system's physical memory which is 8GB gets fully consumed (System got stuck) , What are the possible ways to merge these files , and later how can I effectively use that single file(biggest one). Here is the code through which I am merging those qvds

[Incremental]:

NoConcatenate

Load *;

Load * from

'..\Data Files\File1.qvd'(qvd);

Concatenate

Load * from

'..\Data Files\File2.qvd'(qvd);

Concatenate

Load * from

'..\Data Files\File3.qvd'(qvd);

Concatenate

Load * from

'..\Data Files\File4.qvd'(qvd);

Concatenate

Load * from

'..\Data Files\File5.qvd'(qvd);

Concatenate

Load * from

'..\Data Files\File6.qvd'(qvd);

STORE [Incremental] into '..\Data Files\File.qvd'(qvd);

[kaushiknsolanki]

Hi,

     It doesnt matter whether you are using single qvd of 10gb or 5 Qvd of 2 gb each.

     When ever you load the data from qvd, all data will be moved to RAM and then other calculations will be done if any.

     So the only option i see is you increase the configuration of your computer.

Regards,

Kaushik Solanki

[#3]

Hi,

I would suggest to load only the required fields that are useful into QVD's.

If you are doing it already, then follow kaushik's suggestion .

Regards,

Amay

[#4]

Hi ,

So what will be the good amount of RAM for 10 GB of qvd. And is there any way to use disk instead of RAM?

[kaushiknsolanki]

Hi,

     Ita always better to think about future say 2 years ahead and then estimate the RAM.

     So if assuming every year you will add 10gb of data then you will need almost 30 to 40 GB of RAM.

     so you can plan for 32 GB of RAM.

Regards,

Kaushik Solanki    

[Anonymous]

Hi

               Firstly these 6 QVDs should have some common ground on which to use them as primary keys.

Considering this, you will joins has transformed everything into a single data source.

example:

Concatenate:

Load * from

'..\Data Files\File1.qvd'(qvd);

Left Join (Concatenate)

Load * from

'..\Data Files\File2.qvd'(qvd);

Left Join (Concatenate)

Load * from

'..\Data Files\File3.qvd'(qvd);

leftJoin(Concatenate)

Load * from

'..\Data Files\File4.qvd'(qvd);

left Join(Concatenate)

Load * from

'..\Data Files\File5.qvd'(qvd);

Leftjoin(Concatenate)

Load * from

'..\Data Files\File6.qvd'(qvd);

Store Into Concatenate [..\Data Files\QVDTransformed.qvd];

Regards

shivaji