24 Replies Latest reply: Aug 18, 2015 12:20 PM by Guzman Garcia RSS

    Server can't use two CPU, the performance is very poor

    Guzman Garcia

      Hi all,


      I have a problem with hardware in server, I appreciate if someone can help me.

      My problem is when use two CPU in a dedicated qlikview server, when the server works with 2 CPU the performance is very very poor.


      We have the following hardware in the server:

      • CPU: 2 x Intel® Xeon® E5-2697 v2 (2.7GHz/12-core/30MB/8.0GT-s QPI/130W, DDR3-1866, HT, Turbo2- 3/3/3/3/3/3/3/4/5/6/7/8)
      • RAM: 128Gb RAM PC3-12800R (DDR3-1600)
      • Hard Disk: 2 x HP 300GB 6G SAS 10K 2.5in SC ENT HDD
      • The servers have 2 cpu sockets, numa capability and hyper-threading capability
      • Each CPU is 12 real cores and 12 virtual cores if HT is enabled

       

      When the server works with a single CPU, its work fine. In this scenario the performance with HT enabled is better than HT disabled.

      The problem appears when we add the additional CPU, both CPU are exactly the same model, both were purchased at same time with the server. In this moment the performance of server is very very poor and is impossible that the users uses the application.

       

      What we tried?


      TEST 1:

          • 2 CPU
          • NUMA (In BIOS appears as Node Interleaving) - Disabled
          • Hyperthreading - Enabled

       

      TEST 2:

          • 2 CPU
          • NUMA (In BIOS appears as Node Interleaving) - Enabled
          • Hyperthreading - Enabled


      TEST 3:

          • 2 CPU
          • NUMA (In BIOS appears as Node Interleaving) - Disabled
          • Hyperthreading - Disabled


      TEST 4 (Actual State):

          • 1CPU
          • NUMA (In BIOS appears as Node Interleaving) - Disabled
          • Hyperthreading - Enabled

       

      In the TEST 1, 2 and 3, the performance is poor. But in the TEST 4 (Now) the performance is better, but we need to use two cpu.


      In all test the QlikView Soft-Switch is disabled (If I try to enabled the Soft-Switch the Qlikview Server Service cant start).

       

      Environment:

      • We dont use shared folders outside the server. All the files of the qlikview solutions are local in the server.
      • This is not a virtual environment and the server is dedicated for Qlikview.
      • Server version installed is: 11.0.11440.0
      • We tried installing the QV Server 11.20 SR6 but we have errors in the log, and we came back to the same preovious version (11.0.11440.0) that work fine.

       

      Upcoming Test:

      We will try to test the actual state but with the other cpu to discard a physical hardware issue related with this cpu or the second socket.

       

      Please can someone help me?

      Thank you very much in advance.

      Regards,

      Guzmán.

        • Re: Server can't use two CPU, the performance is very poor
          Adrian Parker

          Guzmán


          Qlik scalability centre in Lund can validate your hardware configuration/config for you.

          Raise a support call and request this.


          Adrian



          • Re: Server can't use two CPU, the performance is very poor
            Hampus von Post

            Hi Guzmán Garcia,

             

            First of all the hardware selected is recommended for QlikView and should produce excellent results.

             

            When you added the second CPU did you make sure to distribute the RAM modules evenly across both CPU slots?

            Refer to the vendor manual, but something often called "hemisphere mode" is crucial for in memory applications.

             

            If the RAM is distributed as per recommendations and both CPUs have the same amount (64 GB) i would try and run some general benchmark on the server. Is it only QlikView that is slow or is a hardware issue?

            Stream is such a memory bandwidth test that could be worthwhile.

            Note: We haven't tested the tool and it is not developed by QlikView, but it is a common tool to measure memory bandwidth in servers.

            http://www.cs.virginia.edu/stream/ref.html

             

            QlikView not starting with NUMA soft switch and not installing correctly are issues I would address with support.

             

            To validate hardware configuration yourself (RAM placement) as well as with vendor. Also try to validate server performance without QlikView.

            Contact QlikView support on the soft switch and installation issues.

            A test on a newer version would be interesting. (such as the SR6).

             

            Regards,

            Hampus von Post

            Scalability Center

              • Re: Server can't use two CPU, the performance is very poor
                Guzman Garcia

                Hampus very thanks for your response.

                 

                When we add the 2 CPU in the server, we distributed RAM memory as you say, for equal between the slots of each CPU socket.

                Also the "hemisphere mode" on the server is "Auto" so if the server detected 2 CPU in sockets and equal distribution of RAM between CPU it should turn on so in my tests this was enabled.

                 

                The hardware apparently works fine. The server dont show any alert. Our problem is that this is a production server with high availability so really is very difficult try a lot. Users should not perceive our test and performance degradation constantly.

                 

                With all that we tested and the configurations that we do, do you think there is any other configurations over the server physically or logically that can affect the performance?

                 

                Thank you very much again.

                Regards,

                Guzmán

                  • Re: Server can't use two CPU, the performance is very poor
                    Oliver Berger

                    Hey,

                     

                    did you cleaned your QV installation? If not, please do it and start the QV services from the ground.

                    Stopp all your QV services. Rename the c:\Programdata\QlikTech Folder and start the re-services.

                    Please check the power settings on your opperation system. It should be 'high performance'

                     

                    You need the QVS / Pub Lizenz Key and the control number as well after this. All your settings are now on default.

                     

                    If you want to rechange this, stopp the services and re-rename the old QlikTech folder in Programdata.

                     

                    /olli

                    • Re: Server can't use two CPU, the performance is very poor
                      Hampus von Post

                      HI Guzmán,

                       

                      Based on what you have stated as far as specification on hardware and configuration there is nothing that would suggest that performance would be anything else that great.

                       

                      In normal cases we talk about three things that effect performance: Environment (hardware really), Application design (data model, expressions etc..) and Usage pattern.

                      If in your case only the environment has changed (one more processor) and performance decrease on a whitelist recommended server then there is something wrong with either hardware or software.

                       

                      I recommend that you log a ticket with support, but it would be interesting to see the results with a newer version of QlikView.

                       

                      A small note; Several small settings can impact the server such as what oliverberger suggested above on "high performance", but none of these are likely to render a well performing one socket go from good to bad. They are however likely to each add a certain amount of percent in performance improvement.

                       

                      Regards,

                      Hampus von Post

                      Scalability Center

                        • Re: Re: Server can't use two CPU, the performance is very poor
                          Guzman Garcia

                          Hi people thank you for your responses,

                           

                          Oliver:

                          When I downgrade QlikView Server version, I uninstalled the previous versions and delete the entry QlikTech folder from ProgramData. Then I did the installation of the current version.

                           

                          Hampus:

                          Today at the end of day, we will test the second CPU (now not used) as single in the same socket used right now. With this test we can discard a CPU problem. We can not test the second CPU socket as single mode, this socket must be use with 2 CPU installed.

                           

                          With the 2 CPU installed on server, we will run hardware diagnostic (RAM, CPU).


                          In relation of our server, what configuration you recommend to us for produce the best performance? I am talking about NUMA (enabled, disabled), HT (enabled, disabled),  Hemisphere mode (Auto, Disabled).

                           

                          About the QlikView Server version:

                           

                          We are thinking to do an upgrade, but this we will in test environment to review the errors in event log (these errors produce low performance and intermittently disconnection).


                          These errors are:

                           

                          1 - 300 Error Restart: Server aborted trying to recover by restart. Reason for restart: Phoenix detected: Performance monitor stuck

                           

                          2 - 300 Error qvpx: Exception while handling request

                           

                          About 1 :

                          1.1: The time duration for Performance logging in the Management Console is 60 minutes.

                          1.2: The CPU and RAM usage is normal. I have enough CPU/RAM resource on the QlikView Server Machine

                          1.3: The server have not any performance counter enabled in windows.

                           

                          About 2:

                          I will try disabling the performance log.

                           

                          Very thanks again!

                          Regards,

                          Guzmán.

                            • Re: Server can't use two CPU, the performance is very poor
                              Pierce Barber

                              For the Upgrade - I would recommend doing this on the test environment, which you stated was your plan, but also testing the documents extensively as there are a lot of changes/bug fixes from 11.00.11440 to SR7 (which was just released today). If you are having issues after the upgrade in production or test you can engage Support to look into it with you.

                                • Re: Server can't use two CPU, the performance is very poor
                                  Guzman Garcia

                                  Hi Pierce,

                                   

                                  I have already a case opened in Partner Portal about the errors in production over SR6.

                                  Do you know if SR7 fix some bug related with my errors?

                                   

                                  Thank you very much

                                  • Re: Server can't use two CPU, the performance is very poor
                                    Guzman Garcia

                                    Pierce, in addition to upgrade, talking about hardware configuration I would like to obtain your recommendation for today test. In relation of our server, what configuration you recommend to us for produce the best performance? I am talking about NUMA (enabled, disabled), HT (enabled, disabled),  Hemisphere mode (Auto, Disabled).

                                      • Re: Server can't use two CPU, the performance is very poor
                                        Pierce Barber

                                        To answer the first post -

                                         

                                        1 - 300 Error Restart: Server aborted trying to recover by restart. Reason for restart: Phoenix detected: Performance monitor stuck

                                         

                                        2 - 300 Error qvpx: Exception while handling request

                                         

                                        These are not just specific to a build (SR) and normally are associated with the environment. The case you have open should give you more information on those two errors, but it should not degrade the performance. That might stem from document development (differences between the large amount of Service Releases you are upgrading from). If that is the case you will need to review the applications and update them to 11.20 SR6/SR7 standards. This can be talked about in the case with the Support Technician assigned to it.

                                         

                                        To answer the second post -

                                         

                                        Recommendation - I would review the articles and items I have sent to you prior, along with the recommendations you have seen in this thread. As stated, different settings can have different effects on an environment, as you've noticed. You might have to fiddle with items to get the top performance for any given machine.

                                         

                                        We can have this topic moved back to the case, unless someone on the forums have any other ideas on why you have such poor performance after you put in the 2nd CPU. This will most likely be determined by an issue with the Hardware itself or possible issue with the Qlikview Version you're running.

                                    • Re: Re: Server can't use two CPU, the performance is very poor
                                      Hampus von Post

                                      Hi Guzmán,

                                       

                                      I would use these settings on the E5-2600 v2 architecture:

                                      HyperThreading Enabled

                                      NUMA disabled (though it doesn't matter with newer QlikView versions)

                                      Hemisphere Auto

                                       

                                      Regards,

                                      Hampus von Post

                                      Scalability Center

                                        • Re: Server can't use two CPU, the performance is very poor
                                          Guzman Garcia

                                          Hi Hampus and Pierce,


                                          In the CPU test, we checked that both CPUs purchased are fine. Each CPU was test independent in server, and work fine.

                                           

                                          We will use the contingency server (same hardware resources than production server) and we will try all the configurations here. We will install the QV Server 11.20 SR7 and try with all configurations in BIOS.

                                           

                                          I will talk to you after this test.

                                          Thank you

                                          Regards

                                            • Re: Re: Server can't use two CPU, the performance is very poor
                                              Guzman Garcia

                                              Hi all,

                                               

                                              Well... Now we have a server (exactly same of production server but with 2 CPU) and in it we can do all test.

                                               

                                              The current server state is the following:

                                              • 2 CPU installed
                                              • QV Server Version: 11.20 SR7
                                              • Performance logging: Disabled
                                              • Hyper Threading: Enabled
                                              • NUMA: Enabled
                                              • Hemisphere Mode: Auto
                                              • RAM Distribution: 64 GB RAM each CPU

                                               

                                              With this settings the performance is lower than the following server state:

                                              • 1 CPU installed
                                              • QV Server Version: 11:20 SR7
                                              • Performance Logging: Disabled
                                              • Hyper Threading: Enabled
                                              • NUMA: Disabled (and Enabled too)
                                              • Hemisphere Mode: Auto

                                               

                                              How I can try to improve the performance to use 2 CPU in the server?

                                              Someone have any suggestion?

                                               

                                              Thank you very much.

                                              Regards,

                                              Guzmán.

                                                • Re: Re: Server can't use two CPU, the performance is very poor
                                                  Oliver Berger

                                                  Ahoi,

                                                   

                                                  did you have some regular benchmark values with one and/or two CPU?

                                                  Why dou you think this is QlikView related?

                                                   

                                                  check

                                                  STREAM Benchmark

                                                  SPECint_rate_base2006

                                                   

                                                  Hemisphere Mode .. sounds like a FUJITSU Server. Please tell me more about it. Try to disable Interleaving and/or Hemisphere mode.

                                                   

                                                  /olli

                                                    • Re: Server can't use two CPU, the performance is very poor
                                                      Guzman Garcia

                                                      Hi Oliver thank you for your response.

                                                       

                                                      I think this problem is QlikView related for the following reasons:

                                                       

                                                      • This server is in the Whitelist QlikView Servers
                                                      • We purchase 2 servers with the same resources, and the problem continue in both servers
                                                      • Is difficult to have a hardware problem and there is no warning,  and more difficult that this problem occurs in the two servers
                                                      • QlikView have know problems with NUMA configurations and hyper-threading feature, Qlikview have problems with these technology
                                                      • Qlikview includes switch-soft tool to avoid NUMA enabled option for example

                                                       

                                                      We have not benchmark values for the server, and we want to try with the STREAM Benchmark, can you tell me where I can download it for Windows Server 2008 R2? I cant find.

                                                       

                                                      We will try disabling the features you talk.

                                                      Thank you very much.

                                                      Regards,

                                                      Guzmán.

                                                        • Re: Server can't use two CPU, the performance is very poor
                                                          Hampus von Post

                                                          Hi Guzmán,

                                                           

                                                          I suggest you continue the dicussion with QlikView support.

                                                           

                                                          A note on the notion that QlikView "have known problems with NUMA.. and hyperthreading" though.

                                                          This statement is incorrect.

                                                          QlikView benefits from HyperThreading and does not have problems with NUMA as the software is NUMA aware.

                                                           

                                                          Regards,

                                                          Hampus von Post

                                                            • Re: Server can't use two CPU, the performance is very poor
                                                              Guzman Garcia

                                                              Hi Hampus thank you for your response.

                                                               

                                                              I said that QlikView have know problems with NUMA because I read some Qlik Whitepapers (as "Impact of NUMA on QV 11 Performance") and I found some sentences as the following:

                                                               

                                                              • "However, extensive tests at our Scalability Center prove that NUMA-enabled servers have a negative impact on QlikView Server performance...Therefore, for NUMA architectures, it is beneficial to disable NUMA in the BIOS settings of the hardware running the QlikView Server in order to achieve optimum performance."


                                                              • "With this set of fairly demanding tests, we conclude that there are differences between the settings configured using the soft-switch in the settings.ini and/or NUMA enabled BIOS. The worst possible performance is received by enabling NUMA and disabling soft-switch. Please take note that if NUMA is enabled on the server, it can be ignored by QlikView by enabling the soft-switch."

                                                               

                                                              One more question: I have opened a support case in Qlik Partner Portal, and they referred me to here, Qlik Scalability Center to solve my problem; and when I saw your signature I thought that you are the Qlik Support to Scalability cases. The Qlik Support is only through Partner Portal and not here?

                                                               

                                                              Thank you very much for your help again!

                                                              Regards,

                                                              Guzmán.

                                          • Re: Server can't use two CPU, the performance is very poor
                                            Roberto Medellin

                                            Hi Guzman.

                                            I have the same question as mgdpuro1, any updates?

                                            We also are suffering performance issues. But we haven't tested using only one CPU.

                                            We have 2 Xeon E5 2560 v2. And we get errors, and also get the feel that it should be faster.

                                             

                                            Regards.

                                              • Re: Server can't use two CPU, the performance is very poor
                                                Guzman Garcia

                                                Hi Roberto and mgdpuro1,

                                                 

                                                I could not resolve the issues with the use of two processors. I contacted to Qlik support but I can't arrive a solution.

                                                If can you try with only one processor, you should see an improvement of performance, the problem is when you are using two processor and the communication between both cpu and the distributed RAM memory.

                                                For any reason, this produces a very poor performance and some errors while users use qv applications.

                                                 

                                                I continue with this issue yet and now my client buy another server with only one processor but with 32 cores. This is better but using only one processor, more than one in a unique server produce poor performance.

                                                 

                                                What is the size of your QVW file? You have reviewed the qv application? (data model, relations of the tables, normalization, use of linked tables, amount of records, etc, etc? Can you split the file in 2 documents for example, one for an aggregation information (with less size) and another with the detail? Or split the history in a file and in another file the last X years/month to use with more frequency?

                                                 

                                                Theses changes can be help you to improvement the performance and usability of application.

                                                 

                                                I hope can help you.

                                                Let me know if you have any question.

                                                Regards,

                                                Guzmán.