Deployment & Management

[Sazabi]

Hello fellow Qliksters,

We have been seeing slow performance on our Qlik servers that are in a 4-socket configuration with 40 cores. 

Looking at the CPU in task manager over the weeks, it always seems like only the first 10 cores are doing something/maxed out. App reloads often time-out and are slow from the front-end perspective.

Are these symptom for something known that can be tweaked?  Why aren't more cores being utilized fully? If I have 30 app reloads running at the same time, shouldn't 30 cores be busy? To me it seems like the server is trying to reload the 30 apps in the 10 cores, getting bottlenecked, and experiencing reload task abortion. (but that's just a guess)

An app reload on this server takes roughly 4x longer to reload than the same app on a smaller, but still saturated server. 

In comparison, one of our smaller 2-socket, 20 core servers has all 20 cores busy. App performance is very good in terms of reload speed and front-end user experience.

Thoughts?

S

[Levi_Turner]

At the outset, 4 socket servers are going to be less performant, as a rule, than 2 socket servers. Fewer sockets means few hops between sockets for multi-threaded operations.

> smaller 2-socket, 20 core servers has all 20 cores
> 4-socket configuration with 40 cores

What are the specific chips in use here? Are they on the recommendation list? Have we reviewed the BIOS settings on the server(s) to ensure similarity?

> Are these symptom for something known that can be tweaked?

First, definitely make sure the performance plan is set to high performance: https://support.qlik.com/articles/000031649

Second, what are the number of concurrent reloads (QMC > Schedulers > Edit > max concurrent reloads)? The high water mark is N-2 where N is the number of logical processors. You're likely to need to do a few trial runs with this because simply maxing things out does get more apps reloading, but it doesn't necessarily optimize for reload time. During operations some requests multi-thread so having some logical processors available is ideal.

[Sazabi]

Hi Levi,

>First, definitely make sure the performance plan is set to high performance:

I can confirm settings are set to High.

Hyperthreading is disabled.

NUMA is disabled.

>Second, what are the number of concurrent reloads?

We have it set to 35

>What are the specific chips in use here? Are they on the recommendation list? 

Intel Xeon E5-4520 v4. Not on the recommendation list as far as I can see.

...which might be the reason.

In the coming weeks we are receiving new servers that are spec'd the same as our 2-socket 20 core machines (Xeon Gold 51xx CPUs, on the whitelist!)

When those new machines arrive, these 4-socket machines will be taken off production and placed into a dev environment for dev work only. I was just wanting to see if we could make any more tweaks so I don't have to dev on slow machines, but this might be as good as it gets because the configuration is way off from the recommendation list?  

S

[Sazabi]

Also,

I was doing some browsing and came across this article: https://community.qlik.com/t5/Support-Knowledge-Base/Hardware-Settings-For-Best-Performance-With-Qli...

>The following settings are recommended to set in BIOS for Qlik Sense:

• Hyper-threading - for most 4-socked solutions having this Enabled is generally beneficial; otherwise it should usually be Disabled

I always thought hyper-threading was to be turned off for better performance? 

S