topic Re: Server can't use two CPU, the performance is very poor in Scalability

Server can't use two CPU, the performance is very poor

Mon, 30 Jun 2014 15:32:49 GMT

Hi all,

I have a problem with hardware in server, I appreciate if someone can help me.

My problem is when use two CPU in a dedicated qlikview server, when the server works with 2 CPU the performance is very very poor.

We have the following hardware in the server:

CPU: 2 x Intel® Xeon® E5-2697 v2 (2.7GHz/12-core/30MB/8.0GT-s QPI/130W, DDR3-1866, HT, Turbo2- 3/3/3/3/3/3/3/4/5/6/7/8)
RAM: 128Gb RAM PC3-12800R (DDR3-1600)
Hard Disk: 2 x HP 300GB 6G SAS 10K 2.5in SC ENT HDD
The servers have 2 cpu sockets, numa capability and hyper-threading capability
Each CPU is 12 real cores and 12 virtual cores if HT is enabled

When the server works with a single CPU, its work fine. In this scenario the performance with HT enabled is better than HT disabled.

The problem appears when we add the additional CPU, both CPU are exactly the same model, both were purchased at same time with the server. In this moment the performance of server is very very poor and is impossible that the users uses the application.

What we tried?

TEST 1:

2 CPU
NUMA (In BIOS appears as Node Interleaving) - Disabled
Hyperthreading - Enabled

TEST 2:

2 CPU
NUMA (In BIOS appears as Node Interleaving) - Enabled
Hyperthreading - Enabled

TEST 3:

2 CPU
NUMA (In BIOS appears as Node Interleaving) - Disabled
Hyperthreading - Disabled

TEST 4 (Actual State):

1CPU
NUMA (In BIOS appears as Node Interleaving) - Disabled
Hyperthreading - Enabled

In the TEST 1, 2 and 3, the performance is poor. But in the TEST 4 (Now) the performance is better, but we need to use two cpu.

In all test the QlikView Soft-Switch is disabled (If I try to enabled the Soft-Switch the Qlikview Server Service cant start).

Environment:

We dont use shared folders outside the server. All the files of the qlikview solutions are local in the server.
This is not a virtual environment and the server is dedicated for Qlikview.
Server version installed is: 11.0.11440.0
We tried installing the QV Server 11.20 SR6 but we have errors in the log, and we came back to the same preovious version (11.0.11440.0) that work fine.

Upcoming Test:

We will try to test the actual state but with the other cpu to discard a physical hardware issue related with this cpu or the second socket.

Please can someone help me?

Thank you very much in advance.

Regards,

Guzmán.

Re: Server can't use two CPU, the performance is very poor

parkera — Mon, 30 Jun 2014 17:01:46 GMT

Guzmán

Qlik scalability centre in Lund can validate your hardware configuration/config for you.

Raise a support call and request this.

Adrian

Re: Server can't use two CPU, the performance is very poor

Tue, 01 Jul 2014 04:36:33 GMT

How can I contact the Scalability Centre by call?

I create a case in Partner Portal with my problem but they referred me to here.

Thank you very much.

Regards

Re: Server can't use two CPU, the performance is very poor

hps — Tue, 01 Jul 2014 07:02:55 GMT

Hi Guzmán Garcia,

First of all the hardware selected is recommended for QlikView and should produce excellent results.

When you added the second CPU did you make sure to distribute the RAM modules evenly across both CPU slots?

Refer to the vendor manual, but something often called "hemisphere mode" is crucial for in memory applications.

If the RAM is distributed as per recommendations and both CPUs have the same amount (64 GB) i would try and run some general benchmark on the server. Is it only QlikView that is slow or is a hardware issue?

Stream is such a memory bandwidth test that could be worthwhile.

Note: We haven't tested the tool and it is not developed by QlikView, but it is a common tool to measure memory bandwidth in servers.

http://www.cs.virginia.edu/stream/ref.html

QlikView not starting with NUMA soft switch and not installing correctly are issues I would address with support.

To validate hardware configuration yourself (RAM placement) as well as with vendor. Also try to validate server performance without QlikView.

Contact QlikView support on the soft switch and installation issues.

A test on a newer version would be interesting. (such as the SR6).

Regards,

Hampus von Post

Scalability Center

Re: Server can't use two CPU, the performance is very poor

Wed, 02 Jul 2014 01:00:38 GMT

Hampus very thanks for your response.

When we add the 2 CPU in the server, we distributed RAM memory as you say, for equal between the slots of each CPU socket.

Also the "hemisphere mode" on the server is "Auto" so if the server detected 2 CPU in sockets and equal distribution of RAM between CPU it should turn on so in my tests this was enabled.

The hardware apparently works fine. The server dont show any alert. Our problem is that this is a production server with high availability so really is very difficult try a lot. Users should not perceive our test and performance degradation constantly.

With all that we tested and the configurations that we do, do you think there is any other configurations over the server physically or logically that can affect the performance?

Thank you very much again.

Regards,

Guzmán

Re: Server can't use two CPU, the performance is very poor

Wed, 02 Jul 2014 07:12:33 GMT

Hey,

did you cleaned your QV installation? If not, please do it and start the QV services from the ground.

Stopp all your QV services. Rename the c:\Programdata\QlikTech Folder and start the re-services.

Please check the power settings on your opperation system. It should be 'high performance'

You need the QVS / Pub Lizenz Key and the control number as well after this. All your settings are now on default.

If you want to rechange this, stopp the services and re-rename the old QlikTech folder in Programdata.

/olli

Re: Server can't use two CPU, the performance is very poor

hps — Wed, 02 Jul 2014 09:01:15 GMT

HI Guzmán,

Based on what you have stated as far as specification on hardware and configuration there is nothing that would suggest that performance would be anything else that great.

In normal cases we talk about three things that effect performance: Environment (hardware really), Application design (data model, expressions etc..) and Usage pattern.

If in your case only the environment has changed (one more processor) and performance decrease on a whitelist recommended server then there is something wrong with either hardware or software.

I recommend that you log a ticket with support, but it would be interesting to see the results with a newer version of QlikView.

A small note; Several small settings can impact the server such as what Oliver Berger suggested above on "high performance", but none of these are likely to render a well performing one socket go from good to bad. They are however likely to each add a certain amount of percent in performance improvement.

Regards,

Hampus von Post

Scalability Center

Re: Re: Server can't use two CPU, the performance is very poor

Wed, 02 Jul 2014 15:02:23 GMT

Hi people thank you for your responses,

Oliver:

When I downgrade QlikView Server version, I uninstalled the previous versions and delete the entry QlikTech folder from ProgramData. Then I did the installation of the current version.

Hampus:

Today at the end of day, we will test the second CPU (now not used) as single in the same socket used right now. With this test we can discard a CPU problem. We can not test the second CPU socket as single mode, this socket must be use with 2 CPU installed.

With the 2 CPU installed on server, we will run hardware diagnostic (RAM, CPU).

In relation of our server, what configuration you recommend to us for produce the best performance? I am talking about NUMA (enabled, disabled), HT (enabled, disabled), Hemisphere mode (Auto, Disabled).

About the QlikView Server version:

We are thinking to do an upgrade, but this we will in test environment to review the errors in event log (these errors produce low performance and intermittently disconnection).

These errors are:

1 - 300 Error Restart: Server aborted trying to recover by restart. Reason for restart: Phoenix detected: Performance monitor stuck

2 - 300 Error qvpx: Exception while handling request

About 1 :

1.1: The time duration for Performance logging in the Management Console is 60 minutes.

1.2: The CPU and RAM usage is normal. I have enough CPU/RAM resource on the QlikView Server Machine

1.3: The server have not any performance counter enabled in windows.

About 2:

I will try disabling the performance log.

Very thanks again!

Regards,

Guzmán.

Re: Server can't use two CPU, the performance is very poor

pbr — Wed, 02 Jul 2014 15:54:28 GMT

For the Upgrade - I would recommend doing this on the test environment, which you stated was your plan, but also testing the documents extensively as there are a lot of changes/bug fixes from 11.00.11440 to SR7 (which was just released today). If you are having issues after the upgrade in production or test you can engage Support to look into it with you.

Re: Server can't use two CPU, the performance is very poor

Wed, 02 Jul 2014 16:15:19 GMT

Hi Pierce,

I have already a case opened in Partner Portal about the errors in production over SR6.

Do you know if SR7 fix some bug related with my errors?

Thank you very much

Re: Server can't use two CPU, the performance is very poor

Wed, 02 Jul 2014 17:19:55 GMT

Pierce, in addition to upgrade, talking about hardware configuration I would like to obtain your recommendation for today test. In relation of our server, what configuration you recommend to us for produce the best performance? I am talking about NUMA (enabled, disabled), HT (enabled, disabled), Hemisphere mode (Auto, Disabled).

Re: Server can't use two CPU, the performance is very poor

pbr — Wed, 02 Jul 2014 17:37:42 GMT

To answer the first post -

1 - 300 Error Restart: Server aborted trying to recover by restart. Reason for restart: Phoenix detected: Performance monitor stuck

2 - 300 Error qvpx: Exception while handling request

These are not just specific to a build (SR) and normally are associated with the environment. The case you have open should give you more information on those two errors, but it should not degrade the performance. That might stem from document development (differences between the large amount of Service Releases you are upgrading from). If that is the case you will need to review the applications and update them to 11.20 SR6/SR7 standards. This can be talked about in the case with the Support Technician assigned to it.

To answer the second post -

Recommendation - I would review the articles and items I have sent to you prior, along with the recommendations you have seen in this thread. As stated, different settings can have different effects on an environment, as you've noticed. You might have to fiddle with items to get the top performance for any given machine.

We can have this topic moved back to the case, unless someone on the forums have any other ideas on why you have such poor performance after you put in the 2nd CPU. This will most likely be determined by an issue with the Hardware itself or possible issue with the Qlikview Version you're running.

Re: Re: Server can't use two CPU, the performance is very poor

hps — Thu, 03 Jul 2014 06:18:56 GMT

Hi Guzmán,

I would use these settings on the E5-2600 v2 architecture:

HyperThreading Enabled

NUMA disabled (though it doesn't matter with newer QlikView versions)

Hemisphere Auto

Regards,

Hampus von Post

Scalability Center

Re: Server can't use two CPU, the performance is very poor

Fri, 11 Jul 2014 15:30:55 GMT

Hi Hampus and Pierce,

In the CPU test, we checked that both CPUs purchased are fine. Each CPU was test independent in server, and work fine.

We will use the contingency server (same hardware resources than production server) and we will try all the configurations here. We will install the QV Server 11.20 SR7 and try with all configurations in BIOS.

I will talk to you after this test.

Thank you

Regards

Re: Re: Server can't use two CPU, the performance is very poor

Fri, 18 Jul 2014 04:29:34 GMT

Hi all,

Well... Now we have a server (exactly same of production server but with 2 CPU) and in it we can do all test.

The current server state is the following:

2 CPU installed
QV Server Version: 11.20 SR7
Performance logging: Disabled
Hyper Threading: Enabled
NUMA: Enabled
Hemisphere Mode: Auto
RAM Distribution: 64 GB RAM each CPU

With this settings the performance is lower than the following server state:

1 CPU installed
QV Server Version: 11:20 SR7
Performance Logging: Disabled
Hyper Threading: Enabled
NUMA: Disabled (and Enabled too)
Hemisphere Mode: Auto

How I can try to improve the performance to use 2 CPU in the server?

Someone have any suggestion?

Thank you very much.

Regards,

Guzmán.

Re: Re: Server can't use two CPU, the performance is very poor

Fri, 18 Jul 2014 08:11:59 GMT

Ahoi,

did you have some regular benchmark values with one and/or two CPU?

Why dou you think this is QlikView related?

check

STREAM Benchmark

SPECint_rate_base2006

Hemisphere Mode .. sounds like a FUJITSU Server. Please tell me more about it. Try to disable Interleaving and/or Hemisphere mode.

/olli

Re: Server can't use two CPU, the performance is very poor

Sat, 19 Jul 2014 23:38:37 GMT

Hi Oliver thank you for your response.

I think this problem is QlikView related for the following reasons:

This server is in the Whitelist QlikView Servers
We purchase 2 servers with the same resources, and the problem continue in both servers
Is difficult to have a hardware problem and there is no warning, and more difficult that this problem occurs in the two servers
QlikView have know problems with NUMA configurations and hyper-threading feature, Qlikview have problems with these technology
Qlikview includes switch-soft tool to avoid NUMA enabled option for example

We have not benchmark values for the server, and we want to try with the STREAM Benchmark, can you tell me where I can download it for Windows Server 2008 R2? I cant find.

We will try disabling the features you talk.

Thank you very much.

Regards,

Guzmán.

Re: Server can't use two CPU, the performance is very poor

hps — Mon, 21 Jul 2014 08:20:23 GMT

Hi Guzmán,

I suggest you continue the dicussion with QlikView support.

A note on the notion that QlikView "have known problems with NUMA.. and hyperthreading" though.

This statement is incorrect.

QlikView benefits from HyperThreading and does not have problems with NUMA as the software is NUMA aware.

Regards,

Hampus von Post

Re: Server can't use two CPU, the performance is very poor

Mon, 21 Jul 2014 12:46:44 GMT

Hi Hampus thank you for your response.

I said that QlikView have know problems with NUMA because I read some Qlik Whitepapers (as "Impact of NUMA on QV 11 Performance") and I found some sentences as the following:

"However, extensive tests at our Scalability Center prove that NUMA-enabled servers have a negative impact on QlikView Server performance...Therefore, for NUMA architectures, it is beneficial to disable NUMA in the BIOS settings of the hardware running the QlikView Server in order to achieve optimum performance."

"With this set of fairly demanding tests, we conclude that there are differences between the settings configured using the soft-switch in the settings.ini and/or NUMA enabled BIOS. The worst possible performance is received by enabling NUMA and disabling soft-switch. Please take note that if NUMA is enabled on the server, it can be ignored by QlikView by enabling the soft-switch."

One more question: I have opened a support case in Qlik Partner Portal, and they referred me to here, Qlik Scalability Center to solve my problem; and when I saw your signature I thought that you are the Qlik Support to Scalability cases. The Qlik Support is only through Partner Portal and not here?

Thank you very much for your help again!

Regards,

Guzmán.

Re: Server can't use two CPU, the performance is very poor

Tue, 22 Jul 2014 08:58:34 GMT

Hi Guzmán,

please have a look into this as well.

please help build a new server - HP DL380 or DL580 | Qlik Community

It seems there is something with sharing ram between CPUs. Check the benchmark tests as follow

/olli