Server can't use two CPU, the performance is very poor

[#1]

Hi all,

I have a problem with hardware in server, I appreciate if someone can help me.

My problem is when use two CPU in a dedicated qlikview server, when the server works with 2 CPU the performance is very very poor.

We have the following hardware in the server:

• CPU: 2 x Intel® Xeon® E5-2697 v2 (2.7GHz/12-core/30MB/8.0GT-s QPI/130W, DDR3-1866, HT, Turbo2- 3/3/3/3/3/3/3/4/5/6/7/8)
• RAM: 128Gb RAM PC3-12800R (DDR3-1600)
• Hard Disk: 2 x HP 300GB 6G SAS 10K 2.5in SC ENT HDD
• The servers have 2 cpu sockets, numa capability and hyper-threading capability
• Each CPU is 12 real cores and 12 virtual cores if HT is enabled
  

When the server works with a single CPU, its work fine. In this scenario the performance with HT enabled is better than HT disabled.

The problem appears when we add the additional CPU, both CPU are exactly the same model, both were purchased at same time with the server. In this moment the performance of server is very very poor and is impossible that the users uses the application.

What we tried?

TEST 1:

• 2 CPU
• NUMA (In BIOS appears as Node Interleaving) - Disabled
• Hyperthreading - Enabled

TEST 2:

• 2 CPU
• NUMA (In BIOS appears as Node Interleaving) - Enabled
• Hyperthreading - Enabled

TEST 3:

• 2 CPU
• NUMA (In BIOS appears as Node Interleaving) - Disabled
• Hyperthreading - Disabled

TEST 4 (Actual State):

• 1CPU
• NUMA (In BIOS appears as Node Interleaving) - Disabled
• Hyperthreading - Enabled

In the TEST 1, 2 and 3, the performance is poor. But in the TEST 4 (Now) the performance is better, but we need to use two cpu.

In all test the QlikView Soft-Switch is disabled (If I try to enabled the Soft-Switch the Qlikview Server Service cant start).

Environment:

• We dont use shared folders outside the server. All the files of the qlikview solutions are local in the server.
• This is not a virtual environment and the server is dedicated for Qlikview.
• Server version installed is: 11.0.11440.0
• We tried installing the QV Server 11.20 SR6 but we have errors in the log, and we came back to the same preovious version (11.0.11440.0) that work fine.

Upcoming Test:

We will try to test the actual state but with the other cpu to discard a physical hardware issue related with this cpu or the second socket.

Please can someone help me?

Thank you very much in advance.

Regards,

Guzmán.

[#2]

Pierce, in addition to upgrade, talking about hardware configuration I would like to obtain your recommendation for today test. In relation of our server, what configuration you recommend to us for produce the best performance? I am talking about NUMA (enabled, disabled), HT (enabled, disabled),  Hemisphere mode (Auto, Disabled).

[pbr]

To answer the first post -

1 - 300 Error Restart: Server aborted trying to recover by restart. Reason for restart: Phoenix detected: Performance monitor stuck

2 - 300 Error qvpx: Exception while handling request

These are not just specific to a build (SR) and normally are associated with the environment. The case you have open should give you more information on those two errors, but it should not degrade the performance. That might stem from document development (differences between the large amount of Service Releases you are upgrading from). If that is the case you will need to review the applications and update them to 11.20 SR6/SR7 standards. This can be talked about in the case with the Support Technician assigned to it.

To answer the second post -

Recommendation - I would review the articles and items I have sent to you prior, along with the recommendations you have seen in this thread. As stated, different settings can have different effects on an environment, as you've noticed. You might have to fiddle with items to get the top performance for any given machine.

We can have this topic moved back to the case, unless someone on the forums have any other ideas on why you have such poor performance after you put in the 2nd CPU. This will most likely be determined by an issue with the Hardware itself or possible issue with the Qlikview Version you're running.

[hps]

Hi Guzmán,

I would use these settings on the E5-2600 v2 architecture:

HyperThreading Enabled

NUMA disabled (though it doesn't matter with newer QlikView versions)

Hemisphere Auto

Regards,

Hampus von Post

Scalability Center

[#5]

Hi Hampus and Pierce,

In the CPU test, we checked that both CPUs purchased are fine. Each CPU was test independent in server, and work fine.

We will use the contingency server (same hardware resources than production server) and we will try all the configurations here. We will install the QV Server 11.20 SR7 and try with all configurations in BIOS.

I will talk to you after this test.

Thank you

Regards

[#6]

Hi all,

Well... Now we have a server (exactly same of production server but with 2 CPU) and in it we can do all test.

The current server state is the following:

• 2 CPU installed
• QV Server Version: 11.20 SR7
• Performance logging: Disabled
• Hyper Threading: Enabled
• NUMA: Enabled
• Hemisphere Mode: Auto
• RAM Distribution: 64 GB RAM each CPU

With this settings the performance is lower than the following server state:

• 1 CPU installed
• QV Server Version: 11:20 SR7
• Performance Logging: Disabled
• Hyper Threading: Enabled
• NUMA: Disabled (and Enabled too)
• Hemisphere Mode: Auto

How I can try to improve the performance to use 2 CPU in the server?

Someone have any suggestion?

Thank you very much.

Regards,

Guzmán.

[#7]

Ahoi,

did you have some regular benchmark values with one and/or two CPU?

Why dou you think this is QlikView related?

check

STREAM Benchmark

SPECint_rate_base2006

Hemisphere Mode .. sounds like a FUJITSU Server. Please tell me more about it. Try to disable Interleaving and/or Hemisphere mode.

/olli

[#8]

Hi Oliver thank you for your response.

I think this problem is QlikView related for the following reasons:

• This server is in the Whitelist QlikView Servers
• We purchase 2 servers with the same resources, and the problem continue in both servers
• Is difficult to have a hardware problem and there is no warning,  and more difficult that this problem occurs in the two servers
• QlikView have know problems with NUMA configurations and hyper-threading feature, Qlikview have problems with these technology
• Qlikview includes switch-soft tool to avoid NUMA enabled option for example

We have not benchmark values for the server, and we want to try with the STREAM Benchmark, can you tell me where I can download it for Windows Server 2008 R2? I cant find.

We will try disabling the features you talk.

Thank you very much.

Regards,

Guzmán.

[hps]

Hi Guzmán,

I suggest you continue the dicussion with QlikView support.

A note on the notion that QlikView "have known problems with NUMA.. and hyperthreading" though.

This statement is incorrect.

QlikView benefits from HyperThreading and does not have problems with NUMA as the software is NUMA aware.

Regards,

Hampus von Post

[#10]

Hi Hampus thank you for your response.

I said that QlikView have know problems with NUMA because I read some Qlik Whitepapers (as "Impact of NUMA on QV 11 Performance") and I found some sentences as the following:

• "However, extensive tests at our Scalability Center prove that NUMA-enabled servers have a negative impact on QlikView Server performance...Therefore, for NUMA architectures, it is beneficial to disable NUMA in the BIOS settings of the hardware running the QlikView Server in order to achieve optimum performance."

• "With this set of fairly demanding tests, we conclude that there are differences between the settings configured using the soft-switch in the settings.ini and/or NUMA enabled BIOS. The worst possible performance is received by enabling NUMA and disabling soft-switch. Please take note that if NUMA is enabled on the server, it can be ignored by QlikView by enabling the soft-switch."

One more question: I have opened a support case in Qlik Partner Portal, and they referred me to here, Qlik Scalability Center to solve my problem; and when I saw your signature I thought that you are the Qlik Support to Scalability cases. The Qlik Support is only through Partner Portal and not here?

Thank you very much for your help again!

Regards,

Guzmán.

[#11]

Hi Guzmán,

please have a look into this as well.

please help build a new server - HP DL380 or DL580 | Qlik Community

It seems there is something with sharing ram between CPUs. Check the benchmark tests as follow

/olli