How-to limit an QV application to number of CPU minutes and RAM usage

[#1]

Does anyone know how to limit an QV application by CPU minutes and also by RAM usage?

I need to define a limit on how long a QV application reload will take on the server, how many CPU seconds/minutes/etc that the QV is allowed to run.

I also need to be able to limit the amount of RAM that the QV application will take from my server.

Scenerio:

1. for a QV application, I set a limit on how long the QV application will run on the server before it gets axed. I set QV application to fail if not completed by 30min of CPU time.

2. for a QV application, I set a limit on how much RAM to allow the QV application. I set to only use 8GB out of 16GB.

3. for a QV application, I set both a CPU and RAM limit.

Server:

- VM hosted Windows 2003 SP2 server 64bit Enterprise

- 16GB RAM, 100GB Hard drive

- QlikView v10 SR1, 64bit

ben

[#2]

Regardless of shared server / individual workstation, there is same question: how to prevent a misbehaved QV script to crash Windows and destroy the NTFS filesystem meanwhile paging-in and paging-out tons of virtual memory?

Example: previously considered "good" script with LOAD * FROM QVD. Than somebody adds a new field to the QVD -> synthetic keys -> virtual memory explodes -> Windows crashes -> Windows does not boot.

Could had been avoided with memory limits? Absolutely !

-Alex

[StefanBackstrand]

I've seen hundreds of Qlikview installations run out of memory because of misbehaving users/applications, but never ever, except in your case, seen Qlikview corrupt NTFS file systems to the degree that the machine does not boot.

Memory limits are most definitely in effect on process level for Qlikview and Qlikview Server. But these limits are in the end solely enforced by the Windows Virtual Memory Manager, which will choose to page memory exceeding the set levels as a part of Windows behavior. I have tried to explain this to you before, Alex. It's not black and white in this case.

But there could be improvements to certain scenarios when it comes to overloaded QVS processes. It's just not as easy as you try to put it.

[#4]

There is no reason to allow a bad reload to fill the virtual memory just for the fun of paging it out. It will fail anyway, but Windows has to page it back to clean up. Paging out 256 GB of physical memory takes an awful lot of time. All other running tasks fail with random errors, corrupting registry, files and disk meanwhile

Think about concrete ways to measure when load is too much. Express it in max sum of QVW filesize, max reloads per day, max concurent reloads per CPU core, max users per QVS, minimum free CPU, minimum free memory, max recipients per distribution task, max bookmarks per .shared file.

Without such limits available, new servers are purchased based on Publisher tasks failure rate, mean time between Windows crashes, and QVS restarts

-Alex

[StefanBackstrand]

Why would Windows want to page 256 GB of RAM? Paging gets done on partial areas of memory, not whole process spaces.

[#6]

Syntetic keys over thousand rows of data finishes and displays a messagebox with warning in QV Desktop.

Syntetic keys over 300 million rows of data in tens of tables never finishes. The fact that there are 256 GB memory makes things worse, because it fails much later than with 1 GB of memory. Fast server disks are able to do much more paging (and damage) than a laptop.

Why isn't there an option in Publisher to stop a reload if there are syntetic keys ? That's because Publisher has no messageboxes, and it is not likely that someone will read the logs of Publisher tasks marked as _SUCCESSFULL_ (in case there is enough memory to finish the syntetic keys).

-Alex

[StefanBackstrand]

Alexandru Toth wrote:Why isn't there an option in Publisher to stop a reload if there are syntetic keys ? That's because Publisher has no messageboxes, and it is not likely that someone will read the logs of Publisher tasks marked as _SUCCESSFULL_ (in case there is enough memory to finish the syntetic keys).

No it's not because Publisher "has no messageboxes", it's because that functionality is not implemented. No one would suggest something as stupid as message boxes in a service. And information and warning messages will of course wind up in the log files for the proper component.

I'm not going to argue about architectural design decisions made by Product, but limiting the usage of certain functionality because some developers build the worlds worst Qlikview applications somehow seems quite stupid, really. It defeats the power of the software.

I don't really know where you're going with this, but if you have suggestions on improving the product, please talk to your Qliktech account manager and they can help receive your idea for improvements, if any, in future releases of Qlikview.

[#8]

In an enterprise environment developers do mistakes, and publisher runs "wrong" tasks. Ideally, the other users and publisher schedules don't need to be affected. There was a question, and two possible ways how to limit memory per task.

I keep wondering why you combat this ideea so agressively .. ?

-Alex

[StefanBackstrand]

I disagree, because you consistently keep making faulty assumptions on how Windows work or does not work, and also seem to totally ignore the fact that many of the issues we saw at your former client was related to either the client environment, badly designed documents or poorly educated developers, I am sorry to say.

Qlikview is not the perfect software, but your tone in our discussions have historically been very negative towards Qlikview and Windows systems in general.

I am sorry to hear that you find this aggressive. I will try and avoid it in the future.

[#10]

So.... while the discussion regarding Windows and QV ability to manage resource intensive applications was an interesting one, it turns out that the practical solution was to limit the applications/documents' timeout so that it doesn't burden the server more than necessary. In this case I chose 60min as that is more than enough time for reload to run (in my opinion and in my environment).

Also, because I have v10 SR1 I chose to limit/throttle the CPU to 90%, and to reduce the affinity to only 6 (of my 😎 CPUs.

These settings allow the applications to complete within a reasonable time, don't swamp the server (as much, still can happen) and allows for a couple CPUs to do other work besides QVS, etc.

AND, I instituded a more strigent review of scripts coupled with a full scale reload test using our QA/Test envionment to catch synthetic keys, select * from tables, etc.

Also, just so you know. The offending applications ran afoul of a small difference between the \\share\directory permissions vs the file systems level permissions. In this case a QVW was storeing a QVD after doing many internal joins, loops and what not. The QVD was not being writen to disk, but it was not giving errors either. Just a 'General Script Error...' within the application logs.

So, I've skirted the real issue, which is that QV could really gain by having more application containment options. While educating developers is a good way to handle some of these issues, without being draconian there is very little way to limit the impact an application or series of applications have on the server. Just something else to put in the Admin enhancement queue.

ben

[#11]

"which is why memory usage needs to be closely monitored and held back"

Could you explain how to tell QV to "hold back" and not try to consume 100% of the server memory?

Ideally Windows should be smarter but it seems it allows up to crash it and we've to restart QV and/or reboot.