Do we have a solution for Working Set Limits? (User Preferences seem ignored)

[evan_kurowski]

*IMPORTANT RANT*

Hello All,

I'm seeking an appropriate way to apply memory 'Working Set Limits'  to desktop sessions.

In QlikView desktop 'User Preferences' it will allow high & low percentage thresholds to be assigned with regards to RAM consumption, but these seem to be completely ignored (and have been seemingly inert for several releases).

Instead, when an inefficient calculation occurs on the UI, the physical memory fills up to capacity then switches over to virtual memory, at which point the machine can slow down to the point of unresponsiveness for extended periods of time.

The natural reaction after staring at a frozen screen for a while, are to force-quit the process or reboot, resulting in the loss of the last round of development changes/reload/etc... (my latest observation was an 8GB physical RAM machine, filled up to 24 GB of virtual memory, in crawling .1GB increments over the space of 20 minutes.  My choice?  Either ditch the development round, or wait it out.)

From descriptions found elsewhere: QV-12-1-Working-Set-Limits
"In my experience, the memory management in QV 12.10 SR3 is pretty much the same as in 11.20 SR16. If a user creates an object which necessitates a large amount of memory to perform the calculations or creates a bad expression resulting in a Cartesian product, etc., Windows may, at its discretion, switch over to paging. This isn't itself a QlikView issue, per se."

If Windows is making decisions about how deeply Qlik processes are allowed to delve into virtual paging memory, those decisions are not making Qlik look good.   It freezes so many sessions the development process becomes abusive, and memory cut-off thresholds need to be applied much earlier.  (there really should be more prominent control available on the Qlik side)

I'm serious... this is a huge issue; stop everything you're doing and develop a fix or dependable mechanism now.  Throughout personal Qlik history this has probably resulted in the destruction or extended disruption of maybe 1 out of 5 development sessions?  It might be THE chronic problem.  (if I'm the only one that got this treatment, then.. that's a jerk move, but... if this is happening to all developers, think about the serious adoption friction!)

Maybe as a developer gets more experienced, they know to be wary, but even then it also creates a constant peril that some inadvertant interaction, a cycle into a large dimension, or a 'clear-selections' is going to activate one of these 'memory bloom booby traps'.


**END RANT**

[Colin-Albert]

Set your user preferences to enable "Save before reload" so if you have created a poorly performing app, then you can crash out of the reload and open the previous version.

The other option is to test the app using Debug mode and apply a load limit to only load say 10 rows from each table to prove the script runs correctly, rather than loading all data.

Setting up the QlikView Desktop Environment
https://community.qlik.com/t5/QlikView-Documents/Setting-up-the-QlikView-Desktop-Environment/ta-p/14...

[Colin-Albert]

You can set a calculation condition on the General Properties of the chart. You can use this to prevent the chart from calculating until data has been filtered to a select a sensible set of records before processing the chart. 

You can customise the "Calculation Condition Unfulfilled" message in the Error Messages box on each chart.

Calculation Condition

Typing an expression in this text box sets a condition that needs to be fulfilled for the chart to be displayed. If the condition is not fulfilled, the text "Calculation condition unfulfilled" will be displayed in the chart. The value may be entered as a calculated formula. Click on the ... button to open the Edit Expression dialog.

This is good practice so a user clicking "Clear" does not have to wait for a large data set to be processed.

[evan_kurowski]

---

@Colin-Albert wrote:

Set your user preferences to enable "Save before reload" so if you have created a poorly performing app, then you can crash out of the reload and open the previous version.

The other option is to test the app using Debug mode and apply a load limit to only load say 10 rows from each table to prove the script runs correctly, rather than loading all data.

Setting up the QlikView Desktop Environment
https://community.qlik.com/t5/QlikView-Documents/Setting-up-the-QlikView-Desktop-Environment/ta-p/14...

---

Hello Colin,

Thank you for your response, it is very appreciated.

"Save Before Reload" is a great feature, I've mused often this setting should be set 'on' by default in new installations.

Reason being I've assisted several new users entering introduction to Qlik development, who get confused as to "why their scripts keep disappearing".   (even as an experienced user, I occasionally get burned by this when jumping across multiple devices onto new Qlik installations, or working with a user who hasn't applied it in their own settings)

A fresh user is prone to syntax errors during scripting, which creates script failures, and without "Save Before Reload" they lose their scripting progress. This frustrating situation has derailed more than one adopter at journey start.

Other considerations:

"Save Before Reload" is not 100% immutable. Somewhere around mid 11.2 SR's, every once in a while I get a script failure that didn't preserve changes. Haven't been able to narrow it to reproducible, but as eyewitness, this slips on occasion.

(In the inverse, 'Save After Reload' sometimes fail to clear the "change indication asterisk" next to the file name in status bar. I'm not sure whether this is related to post-reload triggers? A clean 'Save After Reload' usually clears the asterisk.)

When working with large .qvw (several GB), 'Save Before/After Reload' can slow development, as each save searches for disk space (longest i've experienced has been over 10 minutes per save). Not frequent, but if there's no easy way to reduce data, this is rare circumstance to revert to manual saves.

"Save Before Reload" is not going to safeguard situations where memory bloom happens solely within the UI. A UI design session can go very deep into development changes without involving script reload

(at which point you could invoke 'Save Every N minutes'? "Hmm.. document is pretty big and saving to disk is costing 10 minutes per disk write (1/3 of development time), I'll set it to once per hour. Uh-oh, just lost a session, did anything important happen in that last hour?")

A pivot table with a pair of modestly cardinal dimensions, 100,000 & 300 distinct values, set along each axis, and now the UI object needs 30,000,000 million cells to render and this overwhelms & crashes the device.

'Calculation conditions' are appropriate on a per UI object basis, but during design phase, as syntax evolves & experimentation is underway, creating 'Calculation condition' expressions are not usually the first things being added (does this need to change to a Best Practice? Every time we drop a chart, the first thing to do is set-up safe calculation boundaries?)

How do we gauge the appropriate UI calculation boundaries? Should the limit be a specific number of distinct dimensional values? Combinations of calculations/cells created when incorporating all dimensions? A memory based limit?

Would a universal limit expression be possible, that could be embedded in template and inherited by every instantiation?

(for example: based on # of calculations, without having to name  specific fields)

=If( Me.Calculations() > 10,000,000,Stop())

=If( Me.PhysicalRamUsage() > 95%,Stop())

And there's still the presence of 'Working Set Limits' in the 'User Preferences: General' tab. This indicates an attempt to guard memory at session level, rather than object. How do these settings work, are they active?

(sorry for the lengthy reply, but this has been given some thought and it's more than just random gripe)

[Colin-Albert]

Some good comments there Evan.  Yes "save before reload" can be a PIA when working with large apps - you then need the discipline to manually save 

One thought on a universal Calculation Condition may be to include a "1 as RowCount" field on your fact table, then you could use a simple calc condition such as sum(RowCount) < 10000 as the condition. If your model has multiple fact tables then you may need to add and name the rowcount fields uniquely on each table.

I tend to display a text object with a message such as " 123,456,789 rows selected, please filter data to select fewer than X rows to view chart" instead of using the generic error message. This is shown on top of the chart and hidden using the inverse of the calculation condition logic. The number X is a variable. With this approach users can see the effect their filters are having, rather than the generic "condition unfulfilled" message, and not knowing if their filter has removed 10 rows or 10,000.

Regards the inner working of the Working Set Limits that is beyond my digging into QlikView.

Perhaps   @rwunderlich or @hic  may be able to give more insight on this.

[Colin-Albert]

Oops! I appear to have deleted my reply when I meant to edit it 

You raised some good points Evan - especially regarding "Save before Reload" being a PIA on large apps - then you need to have the discipline to remember to manually save first.
Yes I agree it sould be on by default when installing QV desktop.

Regards calculation conditions - one way to get a universal condition may be to add a field "1 as RowCount" to your fact table. Then you can use a simple text on sum(RowCount) in your calculation condition. 
I prefer to use a text box for the condition message that uses the inverse of the calculation condition to show and hide a message like "YYYY rows selected. Filter the data to select fewer than X rows to view the chart" . This option allows users to see the effect their filters are having on the selected data rather than a bland "condition unfulfilled" message.

As to the details of how the Working Set Limits works on QV desktop - this is beyond my knowledge of the app. 
This may be something that @hic or @rwunderlich can shed more light on, or whether there is a better way than the calc condition to stop the errant application.?

[rwunderlich]

I'll add a few comments. 

WS Limits were never meant to limit the amount of storage a chart could use.  WS Limits are a guide to the QV task as too how much total RAM QV can use before QV has to start managing memory such as throwing out older cache entries.  

With regards to saves and lost work due to a crash.  Yes, you can crash during development.  I do frequent Ctrl-s saves when developing.  A "*" on the title bar indicates you have outstanding changes to save. 

For a too long calculation, I generally can cancel it by pressing the red "X' cancel button on the execution progress window. My experience is that this button is only available if my User Preferences -> Objects -> Calculation Progress Information is set to "Normal".  Or switching to another tab. 

Yes, things can get very wonky if you start using large amounts of virtual storage and in that case it can take a long time to recover even if you can cancel the execution. 

I rarely experience difficulty in this are but I am a very experienced developer.  I do teach many beginners and watch them do crazy things and bump up against bugs causing instability.  I also teach them to do frequent saves 

-Rob
http://masterssummit.com
http://qlikviewcookbook.com
http://www.easyqlik.com

[rwunderlich]

BTW Evan, what QV version are you using?

-Rob

[evan_kurowski]

---

@rwunderlich wrote:

I'll add a few comments. 

---

Hello Rob,

Thanks for responding.  Currently I'm using QlikView 11.2 SR16, & 12.1 SR8.

---

WS Limits were never meant to limit the amount of storage a chart could use.  WS Limits are a guide to the QV task as too how much total RAM QV can use before QV has to start managing memory such as throwing out older cache entries.  

---

When you say 'QV task' I associate this with a QMC task, but 'WS Limit' settings are found under QlikView desktop 'User Preferences', meaning they apply to desktop development.  Do 'WS Limits' affect reloads only?

---

With regards to saves and lost work due to a crash.  Yes, you can crash during development.  I do frequent Ctrl-s saves when developing.  A "*" on the title bar indicates you have outstanding changes to save. 

For a too long calculation, I generally can cancel it by pressing the red "X' cancel button on the execution progress window. My experience is that this button is only available if my User Preferences -> Objects -> Calculation Progress Information is set to "Normal".  Or switching to another tab. 

Yes, things can get very wonky if you start using large amounts of virtual storage and in that case it can take a long time to recover even if you can cancel the execution. 

---

Yes, I agree with frequent saves, I save with an almost paranoid frequency.  It mitigates the potential for loss. 

In this latest instance, the memory seize was caused by accidentally drag selecting too many values in a large cardinality dimension that I was trying to test as a chart dimension.  (But I would swear that there are anomaly events, times this can happen with even no changes at all)

I checked my settings and 'Calculation Progress' was set to normal.. in this instance clicking the red 'x' was not an option as the entire screen does that washed out, faded look where none of the UI is refreshing.

---

I rarely experience difficulty in this are but I am a very experienced developer.  I do teach many beginners and watch them do crazy things and bump up against bugs causing instability.  I also teach them to do frequent saves 

-Rob
http://masterssummit.com
http://qlikviewcookbook.com
http://www.easyqlik.com

---

---

We would want to safeguard the journeys of all users at all levels, that would drive adoption.

While I haven't written the Cookbook on Qlik 😉 I'm pretty experienced as well, with a Developer certification and approved proposals to present at Qonnections.  After 8 years as Qlik specialist, I applied to be a Luminary (*without a hint of irony).   I invented Tourna-Kit™ ! (enjoyable tournament brackets for the whole family) 

*If I don't get it, I am blaming this thread and/or @Marcus Summer

[rwunderlich]

When you say 'QV task' I associate this with a QMC task, but 'WS Limit' settings are found under QlikView desktop 'User Preferences', meaning they apply to desktop development.  Do 'WS Limits' affect reloads only?

No, for QV Desktop I meant the entire qv.exe window.   For Server WS limits apply to the Qvs.exe service.

-Rob