topic Re: QS-Uneven RAM Usage in Management & Governance

QS-Uneven RAM Usage

JustinDallas — Wed, 10 Jul 2024 14:09:42 GMT

Hello Folks,

I have a cluster, and it appears that some of the nodes are hitting 97% RAM and some of the nodes are 30% RAM utilization. This pressure and imbalance leads to failed jobs, and sometimes the cluster having to be restarted.

Example

Node 3(5 of 5 services enabled, non-central node): 97% RAM Usage

Node 4(4 of 4 services enabled): 30% RAM Usage

Node 2(4 of 4 services enabled): 59% RAM Usage

What steps should I take to look figure out why the RAM usage is so imbalanced and the system isn't using the RAM on the other servers.

Any help is greatly appreciated.

Re: QS-Uneven RAM Usage

chriscammers — Wed, 10 Jul 2024 14:41:50 GMT

It is important to understand what services are running on each node.

My initial suspicion is that you have conflicts between your user activity and reloads.

Check these things

Do you have your engine services associated with the virtual proxies that your users connect to? This will ensure that you are load balancing your users to all of your nodes
Are your reload tasks running on only one node of the system? You have options to either balance your reload tasks or confine them to one server and then isolate them from your user activity.
How big are your apps? It is possible that you have a very large app using all the ram on the one node.

Performance tuning is complicated because there are many factors to consider but you do have some options.

Re: QS-Uneven RAM Usage

JustinDallas — Wed, 10 Jul 2024 19:16:31 GMT

I looked at the "Max Concurrent Loads" for the "lazy" nodes, and they were at 4. The nodes that were getting overworked were at 11. I have changed all the scheduler nodes to have the same value of 11. I will report back with my findings tomorrow morning.