Skip to main content
Announcements
Have questions about Qlik Connect? Join us live on April 10th, at 11 AM ET: SIGN UP NOW
hic
Former Employee
Former Employee

In a previous blog post I described the internal data tables and the symbol tables. (See “Symbol Tables and Bit-Stuffed Pointers”). These tables constitute the QlikView internal data model. Then there are the state space vectors that keep track of the user’s selections (See “Colors, states and state vectors”).

In addition to these, there are other structures used to calculate the sheet objects. Sometimes a sheet object can use quite a lot of memory, e.g., a chart with many dimensions.

Often you need to ask yourself – “What in this application uses a lot of memory? What can I improve or optimize?” Is it the data model itself or is it the symbol tables? Or is there a chart that uses a lot of memory?”

To get an answer to these questions, you can use the memory statistics tool. Here follows a basic recipe for a memory analysis:

  • Create a memory statistics file from the application you want to analyze (Document Properties -> General -> Memory Statistics). This will export some memory statistics data to a tab separated file.
  • Create a new QlikView document in which you load the created file.
  • Create list boxes for the fields Class, Type, and Subtype.

To understand what these fields display, see the table below.

Id types.png

The Class field tells you whether the memory used is part of the internal database (data tables and symbol tables), the state space (the selections), the sheet objects (volatile structures to calculate the sheet objects), or a variable. The Type and SubType fields give you additional information about where memory is used.

  • Create a drill-down group of the fields Class, Type, and Subtype named ">Type".
  • Create a stacked bar chart with Id as first dimension, >Type as second dimension, and Sum(Bytes) as expression.
  • Sort the chart descending according to y-value.
  • Restrict the chart to show only the first 10 bars.

You should now have a graph similar to the one below.

Memstat Chart.png

In this you can see which objects, tables, or fields are consuming memory in your application. The bars to the left are the ones that use a lot.

Now you can start to optimize!

If most of your memory consumption is made by sheet objects, look at them and see if there is anything you can do. Does this chart have too many dimensions? Is the expression too complex? Do I really need to sort this list box with a very large number of distinct values? Do I need to show it at all?

If most of your memory consumption is due to database symbols, look at whether you can use the autonumber function to make these symbol tables use less memory.

If most of your memory consumption is made by database tables, you might want to remove columns, or aggregate data. Or maybe realize that optimization isn’t worth wile…

One small word of warning: The numbers in the memory statistics analysis should not be trusted too literally – they do not always describe the situation completely. For instance, if a calculation is cached, it will not show up in the memory usage. Further, if an object has not been opened yet, its memory consumption is unknown and will display as zero. But in spite of these shortcomings, the memory statistics function will most of the time help you find where memory is used.

HIC

24 Comments
Anonymous
Not applicable

Nice one HIC. Great..

0 Likes
5,019 Views
mr_barriesmith
Partner - Creator
Partner - Creator

We've been using this as part of our "formal development" process for a while and developers can gain great insight to how to improve the quality of a QVW by analysing as you have stated.

I am noticing on QlikView 11 that RAM use on Task Manager is very different from the MEMory stats data.  Could you reply with some reasons for this; am guessing it is part of QV.exe developer but also caching etc.

Also, does set analysis and alternate states increase state space?

I believe this is another hidden QlikView jewel.  Few other products give this sort of insight to the internal workings and because QlikView does, we create even more elegant solutions - so my message to QlikTech is please keep this functionality healthy.

5,019 Views
Anonymous
Not applicable

Yup. Good question adam. HIC we are waiting for your reply.

5,019 Views
hic
Former Employee
Former Employee

The memory statistics file only shows memory used by the document, whereas the Task Manager also shows RAM used by the program itself and by the cache (the cache does not belong to a specific document - it is common for all documents).

Alternate states will definitely increase the state space. Not sure if this is correctly reflected in the displayed number, though.

HIC

5,019 Views
mr_barriesmith
Partner - Creator
Partner - Creator

I looked at the question and I expect the problem you have is that you have not navigated through all the sheets.

In my experience, you should open the QVW, make a selection, move to the next sheet, push clear, move to the next sheet and make a selection - do this for every sheet.

While you do the above, QlikView will write the RAM usage into the MEMory stats area.  The calculation time of the sheet objects are:

1. Amount of time to evaluate on last click

2. Average time to evaluate

I think the RAM amount is based on the last click.

If you do not activate a sheet object (when a Chart is minimised), QlikView will not evaluate the object and so you will have no MEMory stats... you have to be sure to Navigate through the document in a way that activates all the objects.

5,019 Views
hic
Former Employee
Former Employee

Adam is completely right.

Also, take a look at the "QlikView Optimizer" application delivered with QV 8.5. http://t.co/uEvnaRRi

HIC

0 Likes
5,019 Views
jcarpenter9
Partner - Creator
Partner - Creator

I have a couple of questions the "QlikView Optimizer" application and memory metadata:

1) What do negative calculation times represent? I ran the optimizer against a poor-performing application and some of the objects generated negative calculation times. Some of them also generated extremely high calculation times, which leads to the next question.

2) What is the scope of the memory statistcs that are collected? That is, do the represent activity since the last time the file was saved, or since it was last opened, or since it was created? In the example above we saw calculation times that were longer than we had the file open before exporting the memory file, and it wasn't clear what this represented.

I'll post some screenshots when I get access to the file again.

Thanks,

James

0 Likes
4,081 Views
jcarpenter9
Partner - Creator
Partner - Creator

Here is a screenshot showing negative calculation time. It's associated with a variable containing a complex expression, not an actual object.

neg_calc_time.png

0 Likes
4,081 Views
jcarpenter9
Partner - Creator
Partner - Creator

One more self-reply. I think I found a clue in the QlikView Help for sheet properties, Objects tab. It describes the Memory column this way:

"The time in milliseconds needed for the last recalculation of the object's contents."

That sounds like the scope of memory statistics is only the last recalculation. Is that right?

0 Likes
4,081 Views
hic
Former Employee
Former Employee

It is indeed the last recalculation. But, I would be careful using the calculation times. Or - I wouldn't trust them to 100%. The calculation algorithm was developed in earlier versions, before we improved the caching. When the cache is used, the result of the calculation is fetched much faster than if it were to be calculated. So, I think the calculation times are much too small in the cases when the cache is used.

HIC

0 Likes
4,081 Views