Skip to main content
Announcements
Have questions about Qlik Connect? Join us live on April 10th, at 11 AM ET: SIGN UP NOW
cancel
Showing results for 
Search instead for 
Did you mean: 
woshua5550
Creator III
Creator III

document size problem

Hi all

I got a problem which really confuse me.

I have two .qvw files, one is of size 6M and another one is of size 25M, it's a wonder that the data model  (in memory) of these two files are exactlly the same ! (same tables with same fileds and records)

Structure.png

difference is , file with 25M was loading from excel and doing some data transfer while file with 6M was loading from a qvd file by samplly  "load * "

but I really don't think this would make any difference on qvw document size

experts , please help to explain this phenomenon . thanks in advance

1 Solution

Accepted Solutions
marcus_sommer

Timestamps could require a lot of resources and any change of the numeric value and/or their formatting will have an impact of how may RAM/Storage will be consumed. Here some background to this topic: The Importance Of Being Distinct.

- Marcus

View solution in original post

9 Replies
Anonymous
Not applicable

Hi

That is the difference. Excel loading will take more memory than loading QVD. QVD loading is Compressed format.

Hope it helps!!

Anonymous
Not applicable

QVD is a native Qlik format and can only be written to and read by QlikView. The file format is optimized for

speed when reading data from a script but it is still very compact. Reading data from a QVD file is

typically 10-100 times faster than reading from other data sources.

woshua5550
Creator III
Creator III
Author

Hi Anil

thx for your reply

Excel loading will take more memory than loading QVD, are you sure about this ?

I did a test just now , firstly load from Excel with about 800k records and generate a qvd, save qvw , it takes 15M,

then create a new qvw loading from qvd file , it takes 15M too...

I believe loading from qvd faster than reading from Excel . but they should take same memory when loading finished I suppose.

Anonymous
Not applicable

Yes, Loading from QVD is faster than Excel.

Siva_Sankar
Master II
Master II

Hi Dave Lau,

I recommend to read some of the discussions and reply by some experts on the topic below and it will be very helpful.

You need some patience in reading the below threads, but all of them are from individuals experience and it will answer most of your questions.

Large amount of data (120 million rows) ... anyone got experience?

How to Handel Large data set while loading in qlikview.

*** 6 Weeks in to QV Development, 30 Million Records QV Document and Help Needed!!! ****

How much data can QlikView handle?

Handling Big Fact table. | Qlik Community

BI Review: A few tips for dealing with large QlikView applications

-Siva

Peter_Cammaert
Partner - Champion III
Partner - Champion III

Things you can check in both documents:

  • Number of rows in each table
  • Cardinality of fields, especially those containing Timestamps. Do they sometimes include time values?
  • Compression settings
  • ...

The fact that your experiment with an Excel source file and a QVD produced from this Excel results in documents with the exact same size, leads me to think that you were initially comparing different data sets.

woshua5550
Creator III
Creator III
Author

Yes..there is something different in time values, but rows ,columns and tables are totally the same.

Do you meant that timestamps would take more memory?

marcus_sommer

Timestamps could require a lot of resources and any change of the numeric value and/or their formatting will have an impact of how may RAM/Storage will be consumed. Here some background to this topic: The Importance Of Being Distinct.

- Marcus

Anonymous
Not applicable

Yes, Time Stamp consumes more memory