In-memory technology still holds for large volume of data in Qlikview?

[#1]

Hi,

It is known that QlikView is powered by In-memort technology. So whenever i'm dealing with small amount data it looks fine  and great!

I'm using hive as my data source and say i have some 10gb of data. So whenever i load the data (10gb)from hive to qlikview,does qlikview load whole 10gb of data into its in-memory?In that case it will be eating up the space, how it is managed.

Also it will compress the data after loading into in-memory. So for 10gb of data how much is the compression ratio?

Waiting for your reply.

[jonathandienst]

Hi

First of all, memory is inexpensive compared to a few years ago and is becoming even cheaper, so a server class machine with 50GB - 100GB of RAM is no longer a massive investment, and I have worked on machines with 1TB of RAM.

Secondly, you don't just dump your data source into a QV model. It requires some design to construct a Qlikview model that contains sufficient information so that the required business questions can queried off the data. Only the required information should be loaded, and at the required level of aggregation.

Finally, the Qlikview does not compress the data in memory. It manages it efficiently using symbol tables and bit stuffed pointers (I think that's the term). When the QVW file is saved, QV can optionally compress the data on disk, but it is optimised for performance, not space, when in memory.

HTH

Jonathan

[#3]

Hi, Thanks for clearing the doubts.