Qlik Community

Ask a Question

Qlik Design Blog

All about product and Qlik solutions: scripting, data modeling, visual design, extensions, best practices, etc.

Henric_Cronström

 

Every time you click, the Qlik engine recalculates everything.

 

Everything.

 

A new selection implies a new situation: Other field values than before are possible; other summations need to be made; the charts and the KPIs get other values than before. The state vectors and the objects are invalidated. Everything needs to be recalculated since this is what the user demands.

 

Well, there is of course a cache also – so that the Qlik engine doesn’t have to recalculate something which has been calculated before. So it isn’t quite true that everything is recalculated: If a calculation has been made before, the result is simply fetched from the cache. But it is true that nothing is pre-calculated. There is no need for that. Everything can be done in real-time.

 

The Qlik engine is an on-demand calculation engine.

 

From a principal point, there are two steps in the recalculation of data: The logical inference in the data model, and the calculations of all objects, including sheet labels and alerts.

 

The logical inference is done first. The goal is to figure out which field values in the symbol tables are possible and which records in the data tables are possible, given the new selection. There is no number crunching involved - it is a purely logical process. The result is stored in the state vectors.

 

Think of it as if the selection propagates from one table in the data model to all other tables. Table after table is evaluated and the Qlik engine figures out which values and records are possible, and which are excluded.

 

Propagation.png

 

When the logical inference is done, the Qlik engine starts to evaluate all exposed objects. List boxes and dimensions in charts must be populated and sorted. All expressions – in charts, in text boxes, in labels, in alerts – must be calculated. Objects that are on other sheets, minimized or hidden, are however not calculated.

 

The calculations are always aggregations based on the data records that have been marked as possible by the logical inference engine. I.e., the objects do not persist any data on their own.

 

The calculation phase is usually the phase that takes time – often over 90% of the response time is due to calculations. The calculations are asynchronous and multi-threaded on several levels: First of all, every object is calculated in its own thread. Secondly, in the 64-bit version, many aggregations e.g. Sum() are calculated using several threads, so that a sum in one single object can be calculated quickly using several CPUs.

 

Finally, when an object has been calculated, it is rendered. Since the calculation is asynchronous and multi-threaded, some objects are rendered long before other objects are ready.

 

And when an object has been rendered, you can click again. And everything is repeated.

 

HIC

 

PS. All of the above is of course true for both QlikView and Qlik Sense. Both use the same engine.

 

If you want to read more about the Qlik engine internals, see

Symbol Tables and Bit-Stuffed Pointers

Colors, States and State vectors

The Calculation Engine

13 Comments
msteedle
Luminary Alumni
Luminary Alumni

There are some exceptions, but the chart expressions don't calculate in the objects described.

Minimized objects may have caption expressions that calculate, and hidden objects and sheets (necessarily) have visibility expressions calculate.

0 Likes
3,112 Views
sparur
Specialist II
Specialist II

Hello, Michael

thanks for your reply.

but could you please specify what do you mean "hidden objects and sheets (necessarily) have visibility expressions calculate"?

I try to understand, what  happen if I have hidden sheet (with a lot of charts) or hidden charts with standard expressions in the active sheet. Do they calculate or not?

thanks,

Anatoly

0 Likes
3,112 Views
Not applicable

Thanks so much for sharing this hic .  Have had an app which has grown organically over time and is now incredibly complex, and in which the UI performance has slowly degraded over time (death by 1000 cuts, or 1000 expressions, you might say).  This comment solved the mystery for me - I was using a data island to conditionally show/hide dimensions, and this appears to have been causing QV to grind to a halt after every click (there are now about 200 expressions on the main tables).  Just did a quick test, and by interfacing the data island with the table via a variable containing the selected values (using a field trigger to update the variable), the UI is has returned to warp speed.   Thanks a lot.


Henric Cronström wrote:

On logical islands:

...But there is a catch: The first step in a chart evaluation is to generate all combinations between the fields in the expression, and this is a single threaded process...

0 Likes
3,073 Views