topic Re: document size problem in QlikView

document size problem

woshua5550 — Wed, 25 Nov 2020 16:16:04 GMT

Hi all

I got a problem which really confuse me.

I have two .qvw files, one is of size 6M and another one is of size 25M, it's a wonder that the data model (in memory) of these two files are exactlly the same ! (same tables with same fileds and records)

difference is , file with 25M was loading from excel and doing some data transfer while file with 6M was loading from a qvd file by samplly "load * "

but I really don't think this would make any difference on qvw document size

experts , please help to explain this phenomenon . thanks in advance

Re: document size problem

Anonymous — Fri, 09 Jun 2017 04:32:29 GMT

That is the difference. Excel loading will take more memory than loading QVD. QVD loading is Compressed format.

Hope it helps!!

Re: document size problem

Anonymous — Fri, 09 Jun 2017 04:37:09 GMT

QVD is a native Qlik format and can only be written to and read by QlikView. The file format is optimized for

speed when reading data from a script but it is still very compact. Reading data from a QVD file is

typically 10-100 times faster than reading from other data sources.

Re: document size problem

woshua5550 — Fri, 09 Jun 2017 05:18:51 GMT

Hi Anil

thx for your reply

Excel loading will take more memory than loading QVD, are you sure about this ?

I did a test just now , firstly load from Excel with about 800k records and generate a qvd, save qvw , it takes 15M,

then create a new qvw loading from qvd file , it takes 15M too...

I believe loading from qvd faster than reading from Excel . but they should take same memory when loading finished I suppose.

Re: document size problem

Anonymous — Fri, 09 Jun 2017 05:56:54 GMT

Yes, Loading from QVD is faster than Excel.

Re: document size problem

Siva_Sankar — Fri, 09 Jun 2017 06:19:06 GMT

Hi Dave Lau,

I recommend to read some of the discussions and reply by some experts on the topic below and it will be very helpful.

You need some patience in reading the below threads, but all of them are from individuals experience and it will answer most of your questions.

Large amount of data (120 million rows) ... anyone got experience?

How to Handel Large data set while loading in qlikview.

*** 6 Weeks in to QV Development, 30 Million Records QV Document and Help Needed!!! ****

How much data can QlikView handle?

Handling Big Fact table. | Qlik Community

BI Review: A few tips for dealing with large QlikView applications

-Siva

Re: document size problem

Peter_Cammaert — Fri, 09 Jun 2017 08:53:14 GMT

Things you can check in both documents:

Number of rows in each table
Cardinality of fields, especially those containing Timestamps. Do they sometimes include time values?
Compression settings
...

The fact that your experiment with an Excel source file and a QVD produced from this Excel results in documents with the exact same size, leads me to think that you were initially comparing different data sets.

Re: document size problem

woshua5550 — Fri, 16 Jun 2017 08:46:26 GMT

Yes..there is something different in time values, but rows ,columns and tables are totally the same.

Do you meant that timestamps would take more memory?

Re: document size problem

marcus_sommer — Fri, 16 Jun 2017 11:03:13 GMT

Timestamps could require a lot of resources and any change of the numeric value and/or their formatting will have an impact of how may RAM/Storage will be consumed. Here some background to this topic: The Importance Of Being Distinct.

- Marcus

Re: document size problem

Anonymous — Sat, 17 Jun 2017 08:17:35 GMT

Yes, Time Stamp consumes more memory