19 Replies Latest reply: Jun 20, 2013 5:11 AM by Giovanni Tirloni RSS

    How-to limit an QV application to number of CPU minutes and RAM usage

    Ben Atmore

      Does anyone know how to limit an QV application by CPU minutes and also by RAM usage?

      I need to define a limit on how long a QV application reload will take on the server, how many CPU seconds/minutes/etc that the QV is allowed to run.

      I also need to be able to limit the amount of RAM that the QV application will take from my server.

       

      Scenerio:

      1. for a QV application, I set a limit on how long the QV application will run on the server before it gets axed. I set QV application to fail if not completed by 30min of CPU time.

      2. for a QV application, I set a limit on how much RAM to allow the QV application. I set to only use 8GB out of 16GB.

      3. for a QV application, I set both a CPU and RAM limit.

       

      Server:

      - VM hosted Windows 2003 SP2 server 64bit Enterprise

      - 16GB RAM, 100GB Hard drive

      - QlikView v10 SR1, 64bit

       

      ben

       

        • How-to limit an QV application to number of CPU minutes and RAM usage
          Kumar M

          Hi...
          One option would be you can set document reload Timeout seconds in QMC under User Document Reload Schedule.

          • How-to limit an QV application to number of CPU minutes and RAM usage
            Stefan Bäckstrand

            I can answer the three questions from a Qlikview functionality point of view, without proposing different kind of hacks to forcedly kill fully legitimate windows processes that are overloaded because of bad user behavior.

            1. There are timeout limits in Qlikview Server Publisher for tasks, I believe.

            2 & 3. These settings in QVS are not per document, they are server wide and are controlled by the Working Set Limits in the QVS performance settings. There are no way, to my knowledge, to limit the usage of RAM for a task in Publisher today.

            I would generally not recommend to "strangle" the resources to a QVS server on a separate machine, since if defeats the purpose of the multi-threading behavior of QV core functionality. If you need to restrict QV to not interfere with other processes, move the other processes to a different machine.

              • How-to limit an QV application to number of CPU minutes and RAM usage
                Ben Atmore

                thanks Kumar, Alex and Stefan.

                I'll follow up on the timeout limits, and lement the lack of per application limits in order to control QV memory and cpu usage.

                While I think it's great that QV has compression, etc, I find it hard to imagine letting a QV query run away in a server env that has 16GB or more of ram and consume 100% of CPU use. I suspect that given a big enough sandbox you could gaurd against sloppy QV scripts. However, that's not an option for me.

                Good responses, I'll follow up to let you know what happens...

                  • How-to limit an QV application to number of CPU minutes and RAM usage

                    Hi,

                    One of my customer has several machines with 256 GB of memory, and 32 cores . There are developers logged in with Remote Desktop using QV developer tool, and scheduling tasks with Publisher.

                    Unfortunately, it is very frequent that somebody messes up, so one Windows process eats 99% CPU, and 100 Gb of virtual memory. At this point Windows is so slow that administrators can't even login to kill processes. The machine is down for everybody, files are corrupted, and people learn why it is a good idea to keep several versions of the same file .

                    --------

                    Unix and mainframe people discovered in the '70 that it is a GOOD THING to put limits on resources, and kill the few runaway ones.

                    There are not many Windows machines with average uptimes of years. Ever wondered why?

                    -Alex

                      • How-to limit an QV application to number of CPU minutes and RAM usage
                        Stefan Bäckstrand

                        (unfortunately, I seemed to have marked this answer as the solution by mistake - sorry for that)

                        Yes, I know, I was responsible for troubleshooting them from our side. =) When the reason for process runaways is caused by such exceptionally empowered users as developers, then the behavior of these roles needs to be revised and changed, clearly. Killing processes might be a solution, but kills cache and non-persistent data in memory. Qlikview servers do use a lot of memory, and does not clear cache by design, which is why memory usage needs to be closely monitored and held back. I agree that the QVS service sometimes benefit from restart, but if you are pending around ~80-90% RAM usage all the time, one should consider to lighten the load or change the behavior on the machine, that's all I'm saying.

                        I've seen a whole bunch of Windows application servers who had uptime far beyond many hundred days per stretch. Maybe your experience with Windows servers aren't as extensive to have allowed you to meet such systems yet? I've managed IIS web servers (both Windows 2000 Server and Server 2003) that had 400+ days of uptime and just kept on rolling. It was quite a shame to shut them down when we needed to switch the hardware to more redundant platforms.

                        • How-to limit an QV application to number of CPU minutes and RAM usage
                          Rob Wunderlich

                           


                          Alexandru Toth wrote:
                          One of my customer has several machines with 256 GB of memory, and 32 cores . There are developers logged in with Remote Desktop using QV developer tool, and scheduling tasks with Publisher.
                          Unfortunately, it is very frequent that somebody messes up, so one Windows process eats 99% CPU, and 100 Gb of virtual memory. At this point Windows is so slow that administrators can't even login to kill processes. The machine is down for everybody, files are corrupted, and people learn why it is a good idea to keep several versions of the same file .


                          I suggest the problem is that developers should not be working on a shared server, rather they should be making "development mistakes" on workstations. I understand that hardware limitations sometimes force people to use this configuration, but it creates a liability in reliabilty and availability.

                           


                          Alexandru Toth wrote:Unix and mainframe people discovered in the '70 that it is a GOOD THING to put limits on resources, and kill the few runaway ones


                          They did, and they still do. But they are using OS's that are designed for multiusers and provide sophisticated facilities for resource monitoring and capping. Windows is not in the same league. Your solution for killing runaway processes is clever and admirable, but not comparable to the techniques available in those other OS's. In most cases, I think it's more efficent to invest in adaquately sized workstations for developers.

                          All that said, you gotta deal with what you've got. But I would encourage customers to look at building workstations for developers and profiling apps before they get to the server.

                            • How-to limit an QV application to number of CPU minutes and RAM usage

                              Regardless of shared server / individual workstation, there is same question: how to prevent a misbehaved QV script to crash Windows and destroy the NTFS filesystem meanwhile paging-in and paging-out tons of virtual memory?

                              Example: previously considered "good" script with LOAD * FROM QVD. Than somebody adds a new field to the QVD -> synthetic keys -> virtual memory explodes -> Windows crashes -> Windows does not boot.

                              Could had been avoided with memory limits? Absolutely !


                              -Alex

                                • How-to limit an QV application to number of CPU minutes and RAM usage
                                  Stefan Bäckstrand

                                  I've seen hundreds of Qlikview installations run out of memory because of misbehaving users/applications, but never ever, except in your case, seen Qlikview corrupt NTFS file systems to the degree that the machine does not boot.

                                  Memory limits are most definitely in effect on process level for Qlikview and Qlikview Server. But these limits are in the end solely enforced by the Windows Virtual Memory Manager, which will choose to page memory exceeding the set levels as a part of Windows behavior. I have tried to explain this to you before, Alex. It's not black and white in this case.

                                  But there could be improvements to certain scenarios when it comes to overloaded QVS processes. It's just not as easy as you try to put it.

                                    • How-to limit an QV application to number of CPU minutes and RAM usage

                                      There is no reason to allow a bad reload to fill the virtual memory just for the fun of paging it out. It will fail anyway, but Windows has to page it back to clean up. Paging out 256 GB of physical memory takes an awful lot of time. All other running tasks fail with random errors, corrupting registry, files and disk meanwhile

                                      Think about concrete ways to measure when load is too much. Express it in max sum of QVW filesize, max reloads per day, max concurent reloads per CPU core, max users per QVS, minimum free CPU, minimum free memory, max recipients per distribution task, max bookmarks per .shared file.

                                      Without such limits available, new servers are purchased based on Publisher tasks failure rate, mean time between Windows crashes, and QVS restarts

                                      -Alex

                                        • How-to limit an QV application to number of CPU minutes and RAM usage
                                          Stefan Bäckstrand

                                          Why would Windows want to page 256 GB of RAM? Paging gets done on partial areas of memory, not whole process spaces.

                                            • How-to limit an QV application to number of CPU minutes and RAM usage

                                              Syntetic keys over thousand rows of data finishes and displays a messagebox with warning in QV Desktop.

                                              Syntetic keys over 300 million rows of data in tens of tables never finishes. The fact that there are 256 GB memory makes things worse, because it fails much later than with 1 GB of memory. Fast server disks are able to do much more paging (and damage) than a laptop.

                                              Why isn't there an option in Publisher to stop a reload if there are syntetic keys ? That's because Publisher has no messageboxes, and it is not likely that someone will read the logs of Publisher tasks marked as _SUCCESSFULL_ (in case there is enough memory to finish the syntetic keys).

                                              -Alex

                                                • How-to limit an QV application to number of CPU minutes and RAM usage
                                                  Stefan Bäckstrand

                                                   


                                                  Alexandru Toth wrote:Why isn't there an option in Publisher to stop a reload if there are syntetic keys ? That's because Publisher has no messageboxes, and it is not likely that someone will read the logs of Publisher tasks marked as _SUCCESSFULL_ (in case there is enough memory to finish the syntetic keys).
                                                  No it's not because Publisher "has no messageboxes", it's because that functionality is not implemented. No one would suggest something as stupid as message boxes in a service. And information and warning messages will of course wind up in the log files for the proper component.

                                                  I'm not going to argue about architectural design decisions made by Product, but limiting the usage of certain functionality because some developers build the worlds worst Qlikview applications somehow seems quite stupid, really. It defeats the power of the software.

                                                  I don't really know where you're going with this, but if you have suggestions on improving the product, please talk to your Qliktech account manager and they can help receive your idea for improvements, if any, in future releases of Qlikview.

                                                    • How-to limit an QV application to number of CPU minutes and RAM usage

                                                      In an enterprise environment developers do mistakes, and publisher runs "wrong" tasks. Ideally, the other users and publisher schedules don't need to be affected. There was a question, and two possible ways how to limit memory per task.

                                                      I keep wondering why you combat this ideea so agressively .. ?

                                                      -Alex

                                                        • How-to limit an QV application to number of CPU minutes and RAM usage
                                                          Stefan Bäckstrand

                                                          I disagree, because you consistently keep making faulty assumptions on how Windows work or does not work, and also seem to totally ignore the fact that many of the issues we saw at your former client was related to either the client environment, badly designed documents or poorly educated developers, I am sorry to say.

                                                          Qlikview is not the perfect software, but your tone in our discussions have historically been very negative towards Qlikview and Windows systems in general.

                                                          I am sorry to hear that you find this aggressive. I will try and avoid it in the future.

                                        • How-to limit an QV application to number of CPU minutes and RAM usage
                                          Ben Atmore

                                          So.... while the discussion regarding Windows and QV ability to manage resource intensive applications was an interesting one, it turns out that the practical solution was to limit the applications/documents' timeout so that it doesn't burden the server more than necessary. In this case I chose 60min as that is more than enough time for reload to run (in my opinion and in my environment).

                                          Also, because I have v10 SR1 I chose to limit/throttle the CPU to 90%, and to reduce the affinity to only 6 (of my 8) CPUs.

                                          These settings allow the applications to complete within a reasonable time, don't swamp the server (as much, still can happen) and allows for a couple CPUs to do other work besides QVS, etc.

                                          AND, I instituded a more strigent review of scripts coupled with a full scale reload test using our QA/Test envionment to catch synthetic keys, select * from tables, etc.

                                          Also, just so you know. The offending applications ran afoul of a small difference between the \\share\directory permissions vs the file systems level permissions. In this case a QVW was storeing a QVD after doing many internal joins, loops and what not. The QVD was not being writen to disk, but it was not giving errors either. Just a 'General Script Error...' within the application logs.

                                          So, I've skirted the real issue, which is that QV could really gain by having more application containment options. While educating developers is a good way to handle some of these issues, without being draconian there is very little way to limit the impact an application or series of applications have on the server. Just something else to put in the Admin enhancement queue.

                                           

                                          ben

                                        • Re: How-to limit an QV application to number of CPU minutes and RAM usage

                                          "which is why memory usage needs to be closely monitored and held back"

                                           

                                          Could you explain how to tell QV to "hold back" and not try to consume 100% of the server memory?

                                           

                                          Ideally Windows should be smarter but it seems it allows up to crash it and we've to restart QV and/or reboot.