Qlik Sense Server Memory: Affected by users filtering?

[#1]

Hello,

My questions is this:

When users filter data in a Qlik Sense app, does it create a memory drain on the server? I know the more users that are logged on and the more data that is stored in memory, the greater the memory strain, but what happens when users are actively working in the dashboard; filtering the data? Does it use more memory?

What about in cases when users create complicated calculations including "if" statements? Could an end user lock up the server?

Thanks!

[rwunderlich]

Memory will be consumed by:

1. The data model itself,

2. Session (selection state) storage for each user.

3. Cached results as charts are recalculated when a user make selections. Each new filter will generate a new cache entry. This is the primary cause of memory growth as the day goes on. QS should manage the cache to not cause a fatal problem in the server.

4. Temporary (transient) memory required to calculate a given chart (see below).

Users developing visualizations can consume a lot of memory by creating wacky (that's a tech term) charts. Again, QS should manage this so it does not cause a failure in the server.

-Rob

http://masterssummit.com

http://qlikviewcookbook.com

[#3]

Thanks for the reply rwunderlich!

I wonder how QS differentiates between Caching chart recalculations and utilizing transient memory during chart calculation; maybe the cache is part of the transient memory as well?

Anywho, I can across a Qlik paper dated 2011 (and for QlikView not Qlik Sense) titled, "Qlik Enterprise Solutions: QlikView Memory Management and Hardware Guidelines", but I'd hope it's the same?

As you can see below, I've included a portion of that paper, and in Qlik's example, the caching load is very minimal (although the paper doesn't mention the file size, data complexity or user activity). In your experience, does this example look about right? Or do you continue to see significant memory drop as users are filtering? Thanks so much!

[rwunderlich]

I haven't read the paper you referenced, but I'm not sure it's a relevant model for typical day to day operations.  The chart you posted seems to show a stress test with a large amount of robot users iterating a relatively small amount of analysis scenarios (probably one or a few QVWs), over a short period of time.  It's useful stuff and demonstrates some important powers and features of QVS.

For this stress test, I'm not surprised that the cache load is minimal as I expect they are repeating a small amount of analysis scenarios.

Cache is not part of the transient memory I mentioned. "Transient memory" is a term I made up to name the immediate working memory required to calculate a chart.  It's used for a second or two and then released.

The cached results are stored in the more "permanent" cache and are much smaller than the working memory required to calc. For example, I've seen ugly pivot tables that require 800MB to calculate but only seem to use 50k of cache to save the results.

For a typical shop, the workload profile is probably different.  A few hundred users, many dozens of QVWs and each each QVW is used in several dozen analysis scenarios (filters and sheets).  Your time period between server restarts is longer than the stress test. As your week goes on, QVWs get updated resulting in new cache entries for the same analysis scenarios.

The QIX engine (my knowledge is mostly based on QVS, I'm assuming the same applies to QS) is designed to assume that it owns almost all the RAM on the machine and manages it accordingly.  All cached results are saved until the lower working set limit is reached and only then are older cache results released.  So it's norma" to see memory increase and it's not a bad thing.

Perhaps a better paper to understand typical memory usage is the one I've attached here (I think there is a more recent copy, but the information in this copy is still valid).

Returning to your original question, are you trying to gain an understanding for planning or curiosity, or are you experiencing a particular problem or phenomena?

-Rob

http://masterssummit.com

http://qlikviewcookbook.com

[#5]

Thank you again rwunderlich! for your reply. I am trying to plan server load. I guess another way of asking (or maybe a new question) is; when the data is loaded in memory, and a large number of users log on, if I have a 10gb app, is the 10gb being loaded by each user resulting in 100gb of memory used? Or since I loaded the data once on the server as 10gb, it is only 10gb totaly in memory?

[rwunderlich]

The 10gb of data is loaded only once and shared by all users of the app.

Each user will require some amount of memory to track that user's selections, perhaps 50MB in this example. Each active user will require 50 MB. So 10 active users would require 10 * 50MB, in addition to the 10GB shared by all users.

You can get the actual numbers for an app using the free Document Analyzer tool.

Qlikview Cookbook: QV Document Analyzer http://qlikviewcookbook.com/recipes/download-info/document-analyzer/

Edit: I just noted that this question is about Qlik Sense -- not currently supported by Document Analyzer.

-Rob

http://masterssummit.com

http://qlikviewcookbook.com