Server can't use two CPU, the performance is very poor

[#1]

Hi all,

I have a problem with hardware in server, I appreciate if someone can help me.

My problem is when use two CPU in a dedicated qlikview server, when the server works with 2 CPU the performance is very very poor.

We have the following hardware in the server:

• CPU: 2 x Intel® Xeon® E5-2697 v2 (2.7GHz/12-core/30MB/8.0GT-s QPI/130W, DDR3-1866, HT, Turbo2- 3/3/3/3/3/3/3/4/5/6/7/8)
• RAM: 128Gb RAM PC3-12800R (DDR3-1600)
• Hard Disk: 2 x HP 300GB 6G SAS 10K 2.5in SC ENT HDD
• The servers have 2 cpu sockets, numa capability and hyper-threading capability
• Each CPU is 12 real cores and 12 virtual cores if HT is enabled
  

When the server works with a single CPU, its work fine. In this scenario the performance with HT enabled is better than HT disabled.

The problem appears when we add the additional CPU, both CPU are exactly the same model, both were purchased at same time with the server. In this moment the performance of server is very very poor and is impossible that the users uses the application.

What we tried?

TEST 1:

• 2 CPU
• NUMA (In BIOS appears as Node Interleaving) - Disabled
• Hyperthreading - Enabled

TEST 2:

• 2 CPU
• NUMA (In BIOS appears as Node Interleaving) - Enabled
• Hyperthreading - Enabled

TEST 3:

• 2 CPU
• NUMA (In BIOS appears as Node Interleaving) - Disabled
• Hyperthreading - Disabled

TEST 4 (Actual State):

• 1CPU
• NUMA (In BIOS appears as Node Interleaving) - Disabled
• Hyperthreading - Enabled

In the TEST 1, 2 and 3, the performance is poor. But in the TEST 4 (Now) the performance is better, but we need to use two cpu.

In all test the QlikView Soft-Switch is disabled (If I try to enabled the Soft-Switch the Qlikview Server Service cant start).

Environment:

• We dont use shared folders outside the server. All the files of the qlikview solutions are local in the server.
• This is not a virtual environment and the server is dedicated for Qlikview.
• Server version installed is: 11.0.11440.0
• We tried installing the QV Server 11.20 SR6 but we have errors in the log, and we came back to the same preovious version (11.0.11440.0) that work fine.

Upcoming Test:

We will try to test the actual state but with the other cpu to discard a physical hardware issue related with this cpu or the second socket.

Please can someone help me?

Thank you very much in advance.

Regards,

Guzmán.

[parkera]

Guzmán

Qlik scalability centre in Lund can validate your hardware configuration/config for you.

Raise a support call and request this.

Adrian

[#3]

How can I contact the Scalability Centre by call?

I create a case in Partner Portal with my problem but they referred me to here.

Thank you very much.

Regards

[hps]

Hi Guzmán Garcia,

First of all the hardware selected is recommended for QlikView and should produce excellent results.

When you added the second CPU did you make sure to distribute the RAM modules evenly across both CPU slots?

Refer to the vendor manual, but something often called "hemisphere mode" is crucial for in memory applications.

If the RAM is distributed as per recommendations and both CPUs have the same amount (64 GB) i would try and run some general benchmark on the server. Is it only QlikView that is slow or is a hardware issue?

Stream is such a memory bandwidth test that could be worthwhile.

Note: We haven't tested the tool and it is not developed by QlikView, but it is a common tool to measure memory bandwidth in servers.

http://www.cs.virginia.edu/stream/ref.html

QlikView not starting with NUMA soft switch and not installing correctly are issues I would address with support.

To validate hardware configuration yourself (RAM placement) as well as with vendor. Also try to validate server performance without QlikView.

Contact QlikView support on the soft switch and installation issues.

A test on a newer version would be interesting. (such as the SR6).

Regards,

Hampus von Post

Scalability Center

[#5]

Hampus very thanks for your response.

When we add the 2 CPU in the server, we distributed RAM memory as you say, for equal between the slots of each CPU socket.

Also the "hemisphere mode" on the server is "Auto" so if the server detected 2 CPU in sockets and equal distribution of RAM between CPU it should turn on so in my tests this was enabled.

The hardware apparently works fine. The server dont show any alert. Our problem is that this is a production server with high availability so really is very difficult try a lot. Users should not perceive our test and performance degradation constantly.

With all that we tested and the configurations that we do, do you think there is any other configurations over the server physically or logically that can affect the performance?

Thank you very much again.

Regards,

Guzmán

[#6]

Hey,

did you cleaned your QV installation? If not, please do it and start the QV services from the ground.

Stopp all your QV services. Rename the c:\Programdata\QlikTech Folder and start the re-services.

Please check the power settings on your opperation system. It should be 'high performance'

You need the QVS / Pub Lizenz Key and the control number as well after this. All your settings are now on default.

If you want to rechange this, stopp the services and re-rename the old QlikTech folder in Programdata.

/olli

[hps]

HI Guzmán,

Based on what you have stated as far as specification on hardware and configuration there is nothing that would suggest that performance would be anything else that great.

In normal cases we talk about three things that effect performance: Environment (hardware really), Application design (data model, expressions etc..) and Usage pattern.

If in your case only the environment has changed (one more processor) and performance decrease on a whitelist recommended server then there is something wrong with either hardware or software.

I recommend that you log a ticket with support, but it would be interesting to see the results with a newer version of QlikView.

A small note; Several small settings can impact the server such as what Oliver Berger suggested above on "high performance", but none of these are likely to render a well performing one socket go from good to bad. They are however likely to each add a certain amount of percent in performance improvement.

Regards,

Hampus von Post

Scalability Center

[#8]

Hi people thank you for your responses,

Oliver:

When I downgrade QlikView Server version, I uninstalled the previous versions and delete the entry QlikTech folder from ProgramData. Then I did the installation of the current version.

Hampus:

Today at the end of day, we will test the second CPU (now not used) as single in the same socket used right now. With this test we can discard a CPU problem. We can not test the second CPU socket as single mode, this socket must be use with 2 CPU installed.

With the 2 CPU installed on server, we will run hardware diagnostic (RAM, CPU).

In relation of our server, what configuration you recommend to us for produce the best performance? I am talking about NUMA (enabled, disabled), HT (enabled, disabled),  Hemisphere mode (Auto, Disabled).

About the QlikView Server version:

We are thinking to do an upgrade, but this we will in test environment to review the errors in event log (these errors produce low performance and intermittently disconnection).

These errors are:

1 - 300 Error Restart: Server aborted trying to recover by restart. Reason for restart: Phoenix detected: Performance monitor stuck

2 - 300 Error qvpx: Exception while handling request

About 1 :

1.1: The time duration for Performance logging in the Management Console is 60 minutes.

1.2: The CPU and RAM usage is normal. I have enough CPU/RAM resource on the QlikView Server Machine

1.3: The server have not any performance counter enabled in windows.

About 2:

I will try disabling the performance log.

Very thanks again!

Regards,

Guzmán.

[pbr]

For the Upgrade - I would recommend doing this on the test environment, which you stated was your plan, but also testing the documents extensively as there are a lot of changes/bug fixes from 11.00.11440 to SR7 (which was just released today). If you are having issues after the upgrade in production or test you can engage Support to look into it with you.

[#10]

Hi Pierce,

I have already a case opened in Partner Portal about the errors in production over SR6.

Do you know if SR7 fix some bug related with my errors?

Thank you very much