4 Replies Latest reply: Oct 6, 2016 6:07 PM by Peter Cammaert RSS

    QlikView Cluster Poor Performance

    Eamonn O'Brien

      Hi All,

       

           I was wondering if you could help me with a major performance issue that we are currently experiencing with our QlikView cluster server environment. We found that our server performance becomes increasingly slow after nightly reloads until the following happens:

                     - QlikView Management Console becomes very slow to use and begins to time out on simple requests.

      - Users using the QlikView Access Points take a long time to connect to the QlikView servers and often end up with the No Server message appearing.

      - The performance of end user applications becomes unacceptable prompting customer complaints.

       

      This has become very frustrating as the whole idea of upgrading to a cluster environment was to solve the above issues. We can temporarily resolve this issue by stopping and restarting the QlikView Distribution engines but it eventually creeps back when scheduled reloads begin to happen again. It seems that the QlikView distributions services are causing congestion on the server which eventually block all QlikView Service communication until a restart of the services are made. We know it's not our servers as RAM is only 57% used on both servers which issues occur. We are executing all our tasks on EDX schedule system.

       

      Any help on solving this issue would be greatly appreciated.

       

      Thanks again,

      Eamonn

        • Re: QlikView Cluster Poor Performance
          Greg Donnells

          Eamonn:

           

          We recently had some strange behavior as well. It would help if you posted some server demographics (what Server OS, CPUs, RAM, and QV Distribution Service Job Settings)

           

          To fix our issues (hanging document reloads, etc) we had to do the registry fix referred to below...

           

          https://community.qlik.com/thread/21718

           

          Due to a limitation associated with using Microsoft’s COM objects we recommend that you limit the number of QlikView Engines (QlikView Management Console>> System>> Setup>> Distribution Services>> Advanced tab) to a maximum of 9 or the number of processor cores available on the host server -1, whichever is lower.

           

          QV says:

          If you have more than 9 processor cores, and wish to run more Engines, contact Support for information regarding a registry change to the Desktop Memory Heap settings on the server.

           

          We just applied the registry fix, rebooted, and have not had issues since. (our QV servers are x86, Windows server 2012, 48 core, 512GB RAM and clustered)

           

          Greg D.

           

            • Re: QlikView Cluster Poor Performance
              Eamonn O'Brien

              Hi Greg,

               

                    Thanks for getting back to me on this. In brief we have our 2 QVS and 2 QDS (Publisher) services on a cluster made up of 2 physical servers with 512GBs RAM, 40 processors and Windows 2012 r2 OS while our QMC and Web services are on virtual machines. We have configured the Distribution services to run 16 QlikView Engines simultaneously and have also changed our Desktop Memory Heap registry settings from 1024,20480,768 to 1024,20480,2048 on both physical servers (although that's the only change we made in this registry file). We also have 1 CPU unselected in System Performance management settings.

               

              This change did help us eliminate the unexplained job failures that occurred over night but we are still left with the poor performance as explained above. Let me know if you need any more information.



              Thanks,

                • Re: QlikView Cluster Poor Performance
                  Morris Paschall

                  here is my experience on this subject. Hope it helps

                   

                  - QlikView Management Console becomes very slow to use and begins to time out on simple requests.

                  Changing the heap size seems to help this issue. Also, we use virtual machines for the QDS servers and if they get tight on resources (too many tasks running at the same time) the QMC slows down. I imagine the same is true for physical servers.

                  We saw too many document admins causing this issue a few years ago. Check to see that you don't have nested groups as doc admins


                  - Users using the QlikView Access Points take a long time to connect to the QlikView servers and often end up with the No Server message appearing.

                  Check to make sure your Root folder is not the same as the access point folder

                  We found that our storage device (where all of the qvws were stored) was getting choked. We upgrade to a faster server

                  Network issues can cause this problem. Rule those out


                  - The performance of end user applications becomes unacceptable prompting customer complaints.

                  This sounds as if your QVS servers are getting overloaded. Check the performance

                  Could also be network related

                  • Re: QlikView Cluster Poor Performance
                    Peter Cammaert

                    I sincerely hope that in the meantime Eamonn found a solution for his problem. However, since the discussion lingers on, I should mention that I think we're (partially) barking up the wrong tree.

                     

                    It seems odd to me that Eamonn mentions having two physical server, each running both a clustered QVS and QDS. The first thing you do when scaling out is to separate Servers and Publishers because they both compete for the same resources all the time (both RAM and CPU cores). And when these resources get in short supply, the Distribution Service may slow down to a crawl (out of RAM) and effictively make the Server unresponsive, often because of monopolising all cores. Remember that running 16 Distribution Engines does not mean that the other 24 cores won't be involved in the reloads. QVB is a multithreaded application that may unfortunately use as many cores as it deems necessary.

                     

                    IMHO the solution to the performance problem (the heap registry patch fixing the reload failures) may be to install another server, and either use it as a single QDS (Publisher) machine, thereby offloading the other clustered QVS-only machines, or vice versa (two clustered Publisher machines and one QVS).

                     

                    Best,

                     

                    Peter

                     

                    [Edit] Oh and BTW, you can only unselect cores for QDS reloads in the configuration files, not in the QMC (yet).