topic Re: qvd file larger than original csv file after simple load in QlikView

qvd file larger than original csv file after simple load

Thu, 20 Feb 2014 23:09:34 GMT

Hey,

If that possible that I convert the data into qvd file and the size is twice as the original csv file??

The QV version is 11.00.11282.0 SR1

I did not change the script just simply load them. Is this possible?

Today I used one data, which is 395MB, and I cut down the rows to make the data to be 77.2 MB and 84 K. The file 1, 2, 3 have exactly the same columns. and I converted them all into .qvd file. All the qvd files are bigger than .csv file.

	.csv file	.qvd file	x times (.qvd/.csv)
File 1	395 MB	710 MB	1.797468354
File 2	77.2 MB	133 MB	1.722797927
File 3	84 K	252 MB	3

Could anyone let me know why this will happen?

Re: qvd file larger than original csv file after simple load

Siva_Sankar — Fri, 21 Feb 2014 03:20:58 GMT

Hi,

It wont be possible unless it is incremental load. However, it will be easy to identify the issue if you post the script here.

-Siva

Re: qvd file larger than original csv file after simple load

Fri, 21 Feb 2014 03:34:49 GMT

I just open the script editor, click table files and choose the .csv file with label embeded and comma as delimiter. Then I click finish and load the data. I keep the manual process as less as possible in order to eliminate any possible error from me. But the file is still that large as I listed above. That's why I am so confused.

So, in conclusion, qvd file should always be small than csv file?

Re: qvd file larger than original csv file after simple load

Siva_Sankar — Fri, 21 Feb 2014 03:38:09 GMT

Yes, QVDs will be less than the size of its source. Did you try reading the QVDs in separate qlikview file to check whether the same number of records are in QVD and CSV file.

Regards.

Siva

Re: qvd file larger than original csv file after simple load

Anonymous — Fri, 21 Feb 2014 03:45:21 GMT

I don't have an interpretation for this issue, but it seems natural that the qvd files will have bigger size than spread sheets and other flat files like xml. However, qvds have better performance 10 to 100x times faster when loading data from them. I don't think that the Qlikview version is causing this issue. It will happen with any other versions.

Re: qvd file larger than original csv file after simple load

Siva_Sankar — Fri, 21 Feb 2014 03:50:00 GMT

if the source file was small, then the QVD could be larger, due to the XML header in the QVD file. Also, the QVD file size depends on the number of distinct values for each field, if this is large (eg distinct values in every record, and that do not compress well), then the QVD could use at least as much (or more) space as the source file.

Regards.

Siva

Re: qvd file larger than original csv file after simple load

Tue, 25 Feb 2014 22:55:31 GMT

if i replace all the missing value with 0, will the size of qvd be small? I tried to conver the csv to excel and replace all the missing value with 0, the size of new qvd is half of the size of the qvd converted from csv.

Re: qvd file larger than original csv file after simple load

stevedark — Tue, 25 Feb 2014 23:59:34 GMT

It will depend a lot on the uniqueness of the data in each of the columns. As has already been stated there are XML tags in the QVD, along with header information and meta data - all of which add to the size a bit. If you have a file with many repeated values QlikView's algorithms replace the repeated values with pointers and this can cause a large shrinking in the file size. If you have many unique values, say many columns with no duplicates, then there is no reduction in size from duplicates being removed, but you would have the increase from the header and meta data.

If you look at the Table Viewer in QlikView and hover over the tables and fields in there it gives you an idea of the number of unique values in the table.

Hope that helps explain.

Steve