QS-Uneven RAM Usage

[JustinDallas]

Hello Folks,

I have a cluster, and it appears that some of the nodes are hitting 97% RAM and some of the nodes are 30% RAM utilization.  This pressure and imbalance leads to failed jobs, and sometimes the cluster having to be restarted.

Example

Node 3(5 of 5 services enabled, non-central node): 97% RAM Usage

Node 4(4 of 4 services enabled): 30% RAM Usage

Node 2(4 of 4 services enabled): 59% RAM Usage

What steps should I take to look figure out why the RAM usage is so imbalanced and the system isn't using the RAM on the other servers.

Any help is greatly appreciated.

[chriscammers]

It is important to understand what services are running on each node.

My initial suspicion is that you have conflicts between your user activity and reloads.

Check these things

1. Do you have your engine services associated with the virtual proxies that your users connect to? This will ensure that you are load balancing your users to all of your nodes
2. Are your reload tasks running on only one node of the system? You have options to either balance your reload tasks or confine them to one server and then isolate them from your user activity.
3. How big are your apps? It is possible that you have a very large app using all the ram on the one node.

Performance tuning is complicated because there are many factors to consider but you do have some options.

[JustinDallas]

I looked at the "Max Concurrent Loads" for the "lazy" nodes, and they were at 4.  The nodes that were getting overworked were at 11.  I have changed all the scheduler nodes to have the same value of 11.  I will report back with my findings tomorrow morning.