How Is Data Stored In Qlik Sense And QlikView?

[ToniKautto]

Data loaded is structured in a flat table. This table has rows and fields, and all fields have the same number of rows. All the value are stored as dual values by the Qlik Indexing (QIX) engine. 



The distinct values of each loaded field are collected into symbol tables. There is one symbol table per field, and each symbol table has a pointer value assigned for each distinct value. The pointers are unique within each symbol table, but not among all symbol tables. QlikView has a limit of maximum of 2 billion (2^31) symbols per symbol table.

NOTE: The distinctness of values are based on the dual value's underlying numerical value. This means that dual values with a different text value, but identical numerical values will be represented by the same symbol. 

The pointer values are bit-stuffed, which means that they only have as many bits that it takes to represent the number of rows in the symbol table. For example, a field with four distinct values needs a pointer with only two bits length, because that is the number of bits it takes to represent four (2^2) values. In the pictures below the pointers are represented as binary values.

A symbol table can be visualized by presenting the field in a list box since it shows the distinct values in a field. 



The in-memory data table will not store the actual loaded values. In the data table the values are replaced by the pointer values. This allows for significant data compression since the pointers are many times shorter than the actual data value. Data can many times be processed quickly through the pointer values, since they have fewer bits to process compared to the real data value. 




Community blog post on QlikView symbol tables and optimization considerations: http://community.qlik.com/blogs/qlikviewdesignblog/2012/11/20/symbol-tables-and-bit-stuffed-pointers