Qlik Community

New to QlikView

Discussion board where members can get started with QlikView.

Announcements
Announcing the newest addition to the Qlik Community, Qlik Gallery! Learn More
Highlighted
woshua5550
Contributor III

document size problem

Hi all

I got a problem which really confuse me.

I have two .qvw files, one is of size 6M and another one is of size 25M, it's a wonder that the data model  (in memory) of these two files are exactlly the same ! (same tables with same fileds and records)

Structure.png

difference is , file with 25M was loading from excel and doing some data transfer while file with 6M was loading from a qvd file by samplly  "load * "

but I really don't think this would make any difference on qvw document size

experts , please help to explain this phenomenon . thanks in advance

Tags (3)
1 Solution

Accepted Solutions
MVP & Luminary
MVP & Luminary

Re: document size problem

Timestamps could require a lot of resources and any change of the numeric value and/or their formatting will have an impact of how may RAM/Storage will be consumed. Here some background to this topic: The Importance Of Being Distinct.

- Marcus

9 Replies
qlikviewnovice
Valued Contributor II

Re: document size problem

Hi

That is the difference. Excel loading will take more memory than loading QVD. QVD loading is Compressed format.

Hope it helps!!

qlikviewnovice
Valued Contributor II

Re: document size problem

QVD is a native Qlik format and can only be written to and read by QlikView. The file format is optimized for

speed when reading data from a script but it is still very compact. Reading data from a QVD file is

typically 10-100 times faster than reading from other data sources.

woshua5550
Contributor III

Re: document size problem

Hi Anil

thx for your reply

Excel loading will take more memory than loading QVD, are you sure about this ?

I did a test just now , firstly load from Excel with about 800k records and generate a qvd, save qvw , it takes 15M,

then create a new qvw loading from qvd file , it takes 15M too...

I believe loading from qvd faster than reading from Excel . but they should take same memory when loading finished I suppose.

qlikviewnovice
Valued Contributor II

Re: document size problem

Yes, Loading from QVD is faster than Excel.

Siva_Sankar
Honored Contributor

Re: document size problem

Hi Dave Lau,

I recommend to read some of the discussions and reply by some experts on the topic below and it will be very helpful.

You need some patience in reading the below threads, but all of them are from individuals experience and it will answer most of your questions.

Large amount of data (120 million rows) ... anyone got experience?

How to Handel Large data set while loading in qlikview.

*** 6 Weeks in to QV Development, 30 Million Records QV Document and Help Needed!!! ****

How much data can QlikView handle?

Handling Big Fact table. | Qlik Community

BI Review: A few tips for dealing with large QlikView applications

-Siva

Re: document size problem

Things you can check in both documents:

  • Number of rows in each table
  • Cardinality of fields, especially those containing Timestamps. Do they sometimes include time values?
  • Compression settings
  • ...

The fact that your experiment with an Excel source file and a QVD produced from this Excel results in documents with the exact same size, leads me to think that you were initially comparing different data sets.

woshua5550
Contributor III

Re: document size problem

Yes..there is something different in time values, but rows ,columns and tables are totally the same.

Do you meant that timestamps would take more memory?

MVP & Luminary
MVP & Luminary

Re: document size problem

Timestamps could require a lot of resources and any change of the numeric value and/or their formatting will have an impact of how may RAM/Storage will be consumed. Here some background to this topic: The Importance Of Being Distinct.

- Marcus

qlikviewnovice
Valued Contributor II

Re: document size problem

Yes, Time Stamp consumes more memory