29 Replies Latest reply: Jul 19, 2014 10:12 AM by Bill Markham RSS

    QVS Memory usage -> Daily restarts of QVS service

    Klaus Feldam

      QV 11.20.12235.0

      Windows Server 2008 R2 SP1

      X64

      Memory: 28GB

      3x Xeon EON E5530 2.4GHz

      10 users

      10 QVW documents

      <500,000 records

      ------------------------

       

      We are daily running through the memory ceiling of our QlikView 11 server, with the QVS consuming 70-80% of available memory. At that point, users are kicked off and unable to reconnect as memory utilization keeps growing. It's not until we stop and restart the QVS service that normal operation is restored. This happens every 18-24 hours.

       

      We only have 10 users and 10 QVW documents. I have previously worked in a 250-user QlikView environment with 8 quad core processors and 128GB memory - same problem.

       

      Does anyone have any recommendations to resolve this problem?

      If there isn't a solution to this unsustainable daily operational issue and memory 'leak' maybe it is time to finally look at a competitor solution.

       

      After 4 years of using QlikView, I love the product, but can't believe Qlik hasn't found a sustainable solution to this memory issue.

        • Re: QVS Memory usage -> Daily restarts of QVS service
          Klaus Feldam

          Three restarts of the QVS service so far today.

          • Re: QVS Memory usage -> Daily restarts of QVS service
            Felim Shanaghy

            I am aware of cases where this has impacted clients to the point of having to revert to QV10 to stop the memory burn..

            • Re: QVS Memory usage -> Daily restarts of QVS service
              Colin Albert

              How frequent are your reloads? It could be  that there is insufficient RAM to support your users accessing the applications, and also running the reload process running during the day.

               

              I would suggest running an app to view your QV logs and see what RAM is consumed when your reloads run without users accessing the applications, and then how much ram is consumed by user access. This may help identify if the issue is in optimising your load scripts, optimising your chart expressions, or even simply changing your reload schedule.

                • Re: Re: QVS Memory usage -> Daily restarts of QVS service
                  Klaus Feldam

                  How frequent are your reloads? It could be  that there is insufficient RAM to support your users accessing the applications, and also running the reload process running during the day.

                   

                  We have 12 QVWs. 5 reload daily. 6 reload every hour (simple dashboards with max three charts that pulls performance metrics) and one (our main QVW) reloads every 30 minutes.  

                   

                  I would suggest running an app to view your QV logs and see what RAM is consumed when your reloads run without users accessing the applications, and then how much ram is consumed by user access. This may help identify if the issue is in optimising your load scripts, optimising your chart expressions, or even simply changing your reload schedule.

                   

                  We are using the Governance Dashboard. I will analyse this closer based on your troubleshooting recommendations.

                  Thank you.

                • Re: QVS Memory usage -> Daily restarts of QVS service
                  Prem Kumar Thangallapally

                  i think after users logout from the session , the memory from the ram is not releasing , once check , active users  when users logged in and all users logged off..

                  • Re: QVS Memory usage -> Daily restarts of QVS service
                    Mark O'Donovan

                    Hi Klaus,

                     

                    If you are using publisher you could look into moving it to a new server if possible.

                     

                    This is what we have done here for free some RAM.

                     

                    Here is some info on how to move publisher:

                     

                    Re: Publisher install in different machine

                     

                    Thanks

                     

                    Mark

                     

                    http://www.youtube.com/practicalqlik

                    • Re: QVS Memory usage -> Daily restarts of QVS service
                      Richard Garforth

                      Klaus,

                       

                      A Couple of ideas, that have helped me.

                       

                      There is a server option to allow only one copy in memory at a time, this needs to be set if you are doing frequent reloads, and users are connected for long sessions.

                       

                      also set the qlikview documents parameter

                      "Refresh Mode when document is Updated on Server" to

                      "Server performs refresh automatically without any client action"

                       

                      Does the increase in memory coincide withthe reload of documents?

                      How much space do the documents take straight after reload without anyone using them?  does the memory utilisation increase rapidly as the users start to use the applicaitons?

                       

                      Richard

                       

                        • Re: QVS Memory usage -> Daily restarts of QVS service
                          Klaus Feldam

                          Thanks, Richard.

                          I have changed the setting to 'allow only one copy in memory' and will see how that affects users, memory utilization and performance.

                           

                          "Refresh Mode when document is Updated on Server" is already set to

                          "Server performs refresh automatically without any client action"

                           

                          The memory utilization increases gradually throughout the day with usage and reloads. We have had increased usage and increases to the number of reloads through the day, so it's difficult to point to which one is the main reason.

                           

                          We have had very little changes to the actual QVWs, but can't rule out ineffective scripts, which will be our next priority.

                           

                          Thanks again for taking the time to respond.

                            • Re: QVS Memory usage -> Daily restarts of QVS service
                              Richard Garforth

                              Klaus,

                               

                              I would like to point you to Henrics document The QlikView Cache  there is a mention in there of how Qlikview manages the cache, and particularly regarding how it caches expressions.  Some of my Applicaitons have similar expressions on charts in various locations, some returning massive  datasets, I think this may be the cause of our issues.  I am going to streamline the expressions in an example app, and see if the memory utilisation will drop in a test environment.  I do not know yet how we will implement this, but I am thinking perhaps devine expressions as variables, and use the variable in the chart/table.

                               

                              Also the cached expressions co exist across applications if the applications share the save qvd and expressions.

                               

                              Richard

                          • Re: QVS Memory usage -> Daily restarts of QVS service
                            Ricardo Gerhard

                            Qlikview is a in-Memory produtc, but exists a lot of config to avoid this issue.

                            I work with Qlikview more than 2 years only in infrastructure scenarios and all tunned servers working fine WITHOUT REBOOT TASKS.

                            The default installation is a poor config. There are a lot of configs in QMC, documents and files.

                            • Re: QVS Memory usage -> Daily restarts of QVS service
                              Rob Wunderlich

                              It's normal for QVS to use 70-80% of memory, as it will fill it with cached results. This is not a leak. However, you mention that users are experiencing instability so somethings not right. Look in the QVS event log to see if there is an error message for the users being terminated or denied access. Also look for log "Working Set..." messages.

                               

                              There were some QVS memory fixes on SR6. Are you using Preload by chance?

                               

                              -Rob

                              • Re: QVS Memory usage -> Daily restarts of QVS service

                                The fact that 70-80% of your memory is consumed by the QVS is absolutely normal. QVS keeps its cache, even if there are no users.

                                 

                                We run an environment with about 250 users and 16 (operational) documents (disk space sizes between 50 Mb and 1.2 Gb) and during the day memory consumption is always around 70-75%. We do, however, not reload during the day, all reloads are done in the morning. Are users typically getting disconnect during the reloads? If so, a useful test would be to lower the working set limits from e.g. 70-90% to 60-80% leaving enough room for the distribution service. Given the high frequency of your reloads, moving the distribution service to another server is also worthwhile considering.

                                 

                                I noticed also you are still on SR2. We also had issues with users getting disconnected in the past (QV9,QV10 and QV11). When analyzing the memory consumption back then, we typically saw high peaks of memory usage, up to 95+ percent during most of these disconnects. For some reason, either there were issues with QVS memory consumption (leaks?) or with the working set management (QV not releasing cache fast enough?). We have had a ticket with QT on this for about 2 years and they never really found a true cause. For some reason, not even clear to the QT-people, the disconnect issues disappeared completely when we upgraded to QV11 SR4. So, you might also consider an upgrade.

                                • Re: QVS Memory usage -> Daily restarts of QVS service
                                  Shane Spencer

                                  Hi Klaus,

                                   

                                  We have similar issues, Qlik have been unable to help and haven't even been able to recreate the issue.

                                   

                                  The problem is a "Memory Contention". When you get to the Working Set Min (default value of 75%) QVS will start to flush cached results. If you try to do something that requires a lot of memory at this point QVS will try to flush memory at the same time as allocating memory to the new task, cpu will run up to 100% causing the server to effectively hang. There are a few things you can do to avoid this.

                                   

                                  Rather than the thrice daily recycle you could implement a cache flush. Update the settings.ini file with

                                  ClearCacheTimesPerDay=1

                                  in the the [Settings 7] configuration section. Or 2 for twice a day, 3 for three times a day etc.

                                  QlikView IS prone to memory leakages, no matter what Qlik would like you to believe, and this works most of the time but sometimes fails to clear the cache.

                                   

                                  Adjust the working set min. At 28GB RAM there's not room for change however with 128GB RAM you can put this higher, maybe 85%. Essentially you need to allow enough memory for all other processes on the server when QVS is at Working Set Min. Use Perfmon to monitor this and pick a suitable level.

                                   

                                  Set all your documents to pre-loaded as this will prevent a document being loaded in to Memory when Memory is at Working Set Min.

                                   

                                  Understand your "document memory footprint". With all you documents pre-loaded restart QVS and see how much memory they use. The rest will be used for cache so it's important to understand how much of your overall memory is used by documents and how much is cached results.

                                   

                                  Understand your reload impact. Use Perfmon to monitor Memory Utilisation during a reload. Isolate the system from users so you can see the impact of just the reload. See how much memory grows during the reload, how long it takes and whether memory drops back to the previous level after. It should drop back but we've experienced "leakages" where after the reload when a document is updated in Memory the old version does not release it's memory. Qlik are still working to fix this but have at least managed to replicate. It may be necessary to move the Publisher to a seperate server as Mark suggested above.

                                    • Re: QVS Memory usage -> Daily restarts of QVS service
                                      Rob Wunderlich

                                      shane_spencer

                                      "Set all your documents to pre-loaded as this will prevent a document being loaded in to Memory when Memory is at Working Set Min."

                                       

                                      I disagree with this recommendation. Instead of responding to user behavior, this artificially drives the RAM usage up. I think it's better to reduce the Document Timeout setting and let unused documents get unloaded sooner.

                                       

                                      -Rob

                                        • Re: Re: QVS Memory usage -> Daily restarts of QVS service
                                          Rob Wunderlich

                                          Klaus,

                                           

                                          Looking at your logs, I think you server is mis-configured or under resourced for how you are trying to use it.

                                           

                                          Here's your memory usage pattern for May 14. Notice how the Avg Loaded Docs keeps increasing. This indicates that you don't have "Allow only one copy of Document in memory" checked on in the server settings (as someone previously recommended). Note that the Doc Settings and Ref Docs to not track the Loaded Docs trend. That's because almost all the Doc Sessions -- 1400 in this case -- are from a user PATHDASH being used from multiple machines. These are short sessions that do not close properly and leave the session and document behind. Is this some kind of automation?

                                           

                                          Some recommendations:

                                          1. Set "Allow only one copy of Document in memory" ON. That should have a lot of positive impact.

                                          2. You say your server is 28MB, but the event logs indicate that min is 16.8 which would be 60%. Was there a reason it was turned down from the default of 70%? I would turn it back up to 70%.

                                          3. Reduce the QEMC "Document Timeout".

                                          4. Examine what's going on with the PATHDASH user. This is not typical user behavior, and may require some adjustment or cleanup.

                                           

                                          -Rob

                                           

                                           

                                           

                                          Memory.jpg

                                            • Re: QVS Memory usage -> Daily restarts of QVS service
                                              Klaus Feldam

                                              Rob,

                                               

                                              Thank you very much for your input and suggestions. You are a great resource on this community.

                                              From the May 14 log, you can see the effects of my QlikView Server Service restarts at 8am and 4pm. The dips are the service restarts.

                                               

                                              1. "Allow only one copy of Document in memory" is already set to ON.

                                              2. I turned the working set down to 60%. I have now set it back to 70%.

                                              3. Reduced the QEMC "Document Timeout" to 120 minutes.

                                              4. The PATHDASH user is used for 4 dashboard monitors that we are using in our office. These monitors have 6-8 Qlikview apps (dashboards) open in a Firefox browser, which rotates through the dashboards every couple of minutes and performs a refresh of the page every time a tab is opened. This explains the number of times PATHDASH opens a document. I haven't found any better way of refreshing my QlikView data on our business dashboards.

                                            • Re: Re: QVS Memory usage -> Daily restarts of QVS service
                                              Shane Spencer

                                              If QlikView was stable / reliable at high memory utilisation you would be right, but it's not. Pre-loading will prevent the documents being loaded in to memory when memory utilisation is already at working set min; from my experience it is this type of activity that requires a large amount of memory that's not readily available that causes QlikView a lot of problems. Therefore it's best to have the documents pre-loaded so you can control when they are loaded in to memory.

                                               

                                              Rob Wunderlich wrote:

                                               

                                              1. Set "Allow only one copy of Document in memory" ON. That should have a lot of positive impact.

                                               

                                              With regards to this setting we've experienced an issues with it not working correctly - a memory leak. What happens is when a document is reloaded a new version will be loaded in to memory but the memory from the old document is not released. We've currently got a case open with Qlik, they've been able to recreate the problem and they're working on a solution but in the short term we've had to implement daily QVS restarts.

                                              • Re: Re: QVS Memory usage -> Daily restarts of QVS service
                                                Richard Garforth

                                                Rob,

                                                 

                                                I think it depends on the circumstance.

                                                 

                                                This setting can be useful when you have a few large documents, rather than many small documents.  For example :- If you expect to run with 3 large documents and you only have 2 loaded, then loading the third when the first 2 have consumed a lot of cache and the system memory utilisation is near working-set min, this can cause problems on some implementations, as the document does not manage to load before it times out the user, since its busy finding memory. (and possibly has much to find)

                                                 

                                                I do agree though if you have 20-30 smaller documents that are infrequently used then allowing unused docs to time out quicker is better.

                                                 

                                                Perhaps the advice fromshane_spencer should have been

                                                 

                                                "if you have large documents that need to be used by many users then preload them to ensure they have memory allocated, if you have small infrequently used documents then do not pre-load"

                                                 

                                                Richard