Qlik Engine: Hypercube Memory Limit

Sonja_Bauernfeind · Nov 11, 2020 4:22:54 AM

The hypercube represents the extraction of data from a Qlik in-memory data model. A typical example of such data is the data needed to display a particular type of visualization object, for instance, a bar chart. Configuration of the contents of a hypercube mainly consists of defining a set of dimensions that should be observed by the hypercube, and a set of measures that should be computed based on those dimensions. The resulting hypercube is a table where each row of the table will contain one unique combination of dimension values along with measures computed as if those dimension values were selected, in the same way as data would be shown in a straight table.

Memory Usage During Calculation

Initially, the Hypercube memory allocation is very small, as it only contains definitions without any actual data. During calculation, the memory allocation for the hypercube varies based on the measured complexity and data volume being processed. This is illustrated by the blue line in the below pictures.

Data for the calculation is extracted from the data model for all the fields that are referred to in dimensions and measures. The number of records required for a calculation depends on the current scope for each measure, which is defined by the current selection, alternate state, and set modifier. The relationship between the fields in the data model is also a factor for the extracted data size, as the records may have to be joined to preserve the logical associated relationship, which may lead to additional records being represented as part of a join of the fields. The complexity of the expression defined in the measure may also affect the memory allocation during a calculation, as the calculation may require partial results to be calculated before the result can be established.

For more insights on how the Engine processes data see below blog posts:

Memory Allocation Check

The Qlik Engine checks the hypercube's memory usage during the calculation. For performance reasons, memory usage is checked periodically rather than on every allocation. The interval between memory checks is not directly time-based, instead, it is based on checkpoints in the execution of the ongoing calculation, meaning that memory allocation is checked when it is logically most suitable. This means that the interval between memory checkpoints might vary over time, which is illustrated by the yellow dots in the below pictures.

Peak Memory (Peak RAM)

The highest measured memory allocation during hypercube calculation is referred to as Peak RAM in the Engine QIX Performance logs. This value is an indication of max memory cost for hypercube calculation and should be considered as an estimate rather than a factual value. The measurement is not accurate, due to the periodical memory allocation check. The actual peak memory allocation is likely to occur between two memory checkpoints, as shown in the below picture.

Qlik provides several tools to enable an analysis of memory utilization for apps or individual objects. These can be used proactively to identify optimization opportunities or reactively for troubleshooting high resource allocation concerns.

Net Memory (Net RAM)

Memory allocation size when the hypercube calculation has finished is referred to as Net RAM in the Engine QIX Performance logs. This measurement indicates the net memory cost for a calculated visualization, which is illustrated with a green dot in the below picture.

net memory chart.png

Hypercube Memory Limit

By default, the Qlik Engine applies a global heuristic to limit the amount of simultaneously executing requests that allocate a lot of memory to calculations. Qlik Engine can be configured to limit hypercube max memory allocation, as referred in Qlik Sense Help for Administrators: Engine configuration

Hypercube calculation is terminated when the memory limit breach has been detected at a memory checkpoint. At this point, the memory allocation is released and, consequently, a visualization depending on the hypercube result set will fail to show an expected result.

This memory limit is evaluated against the most recent memory allocation measurement, which means there is a window in time between memory checkpoints where the memory allocation can go beyond the limit before its detected.
The quicker it is for the Qlik Engine to source new memory from the OS for a request, the higher above the memory limit a request can, theoretically, go before detection. When the OS is able to serve the Qlik Engine with free blocks of memory the allocation speed is much faster compared to when memory needs to be reclaimed from memory currently in use i.e. freeing memory by purging calculation cache results.

Besides the hypercube limit, the engine still has a set of deeper memory throttling functionalities that controls the speed of memory allocation when approaching the configured working set limits, and these are always enabled.

net memory chart.png

Optimization Considerations

Below are some general suggestions on how to approach optimizing memory allocation by governing data model and calculation definitions;

Review affected data model and optimize towards Qlik best practices
- Star or snowflake-shaped data model
- Optimize symbol tables for fields with a high ratio of distinct values, see Qlik Design Blog: The Importance Of Being Distinct as references
- Enumerate text values when applicable, to allow minimizing string-based aggregations
- Govern data model through Qlik data layer, so allow optimized data model in all apps
Optimize app
- Define master items to enable self-service with less risk of poorly defined measures
- Avoid calculated dimensions
- Apply set expression instead of if() statements when applicable
- Avoid string operations when applicable, and utilize numerical operations instead
Evaluate using DisableNewRowApplicator=0 as describe in below support articles, to allow Qlik Engine to alter its calculation logic between columnar or row-based logic based on the data structure.
- QlikView: SE_LOG: when AAALR(1072.471418) is greater than 1.000000, we suggest using new row applicator to imp...
- Qlik Sense: AAALR is registered in the Qlik Sense logs

Qlik Engine: Hypercube Memory Limit