topic Re: How to check if Load Balancing is working for a multi-node Qlik Sense environment? in Management & Governance

How to check if Load Balancing is working for a multi-node Qlik Sense environment?

mwallman — Fri, 26 Apr 2019 12:53:40 GMT

Hello,

We have a two node Qlik Sense environment.

On both nodes we have 5 services running each.

When I go to the Nodes page in the QMC it shows both nodes and the 5 services are running.

When I go to the Engines page it shows both nodes are "running".

However for the last 2-3 days only the central node has been handling actions it seems because the Operations Monitor only registers the central node for CPU, RAM, Concurrent users activities.

Is this normal behaviour?

We have on average about 30 users per day, but being Easter period this week it is averaring around 10 users a day and may only about 2-3 concurrent users.

I am wondering if something is wrong even though thing seem to be "running".

Re: How to check if Load Balancing is working for a multi-node Qlik Sense environment?

Giuseppe_Novello — Fri, 26 Apr 2019 13:59:35 GMT

Have you setup yout Virtual proxy to Load balance to both engines nodes? QMC> virtual proxy> select the Virtual proxy > edit > load balancing> Both Engines nodes are added there?

Gio

Re: How to check if Load Balancing is working for a multi-node Qlik Sense environment?

mwallman — Fri, 26 Apr 2019 16:17:46 GMT

Hi Giuseppe,

Yes both nodes have always been there in the virtual proxy too.

Re: How to check if Load Balancing is working for a multi-node Qlik Sense environment?

Giuseppe_Novello — Fri, 26 Apr 2019 18:18:33 GMT

well then 3 things:

1- You have a load balancing rule that affecting this.

2- User are getting redirect to the wrong proxy that has only central engine linked

3- the proxy linked for the Virtual proxy you are using has blocked the ports for the other engines nodes. ( I am guessing you have central proxy that you are using?)

Gio

Re: How to check if Load Balancing is working for a multi-node Qlik Sense environment?

mwallman — Fri, 26 Apr 2019 19:52:21 GMT

Hi Giuseppe,

1 - We use the default load balancing rules

CentralNode security rule has Resource filter set to *

SecondNode security rule has Resource filter set to App_*

2 - How do we check for this to see if this is a problem?

When I go to Proxies page I can see the following set up:

For the central proxy we have the central proxy and virtual proxy added in virtual proxies.

For second proxy we have the second proxy and the virtual proxy added in the virtual proxies.

3 - How do we check for this?

I am aware that 443, 4244, 4243, 80 ports should not be used by other things? Which ones should I check to see if they are blocked please and how to check if they are blocked?

Re: How to check if Load Balancing is working for a multi-node Qlik Sense environment?

Giuseppe_Novello — Fri, 26 Apr 2019 20:01:53 GMT

#2 - Well it is more a question to you, what URL are you using? and if you sure it is redirecting to the proxy that is linked to a VP that had both engines to load balancing.

#3 - You need 4747 and 4900 and others, read more for further understanding, under "ports examples > multinode" :

https://help.qlik.com/en-US/sense/February2019/Subsystems/PlanningQlikSenseDeployments/Content/Sense_Deployment/Ports.htm

Re: How to check if Load Balancing is working for a multi-node Qlik Sense environment?

mwallman — Fri, 26 Apr 2019 21:01:53 GMT

Hello Giuseppe,

Thanks for replying.

2) We are using HTTPS url, not the IP address but a user friendly name. I can confirm the virtual proxy we are using has both of the nodes added in the load balancing section.

3) I checked if the ports are in listening mode on each node by doing netstat -a -n

From your link I read the "Minimum ports used for communication in multi-node sites" and results from the second node came back as:

4242 - LISTENING

4432 - NOT LISTED

4444 - LISTENING

Other nodes that reported as LISTENING are:

4242

4747

4900

4244

4545

5050

5151

5252

4899

4900

3003

4545

9028

-----------------

We have not changed any configuration on the servers in the previous 30 days, it just seems to strange no activity is happening on the second node. I have checked the server is running, checked Qlik Sense services are running as well.

Re: How to check if Load Balancing is working for a multi-node Qlik Sense environment?

mwallman — Fri, 26 Apr 2019 22:11:35 GMT

Hi Giuseppe,

I noticed that in the second node server there is CPU and RAM being used for the Engine service but I am not sure what for. I saw this while logging into the server and using the Task Manager to see what activity is going on.

For the same time, in the Operations Monitor for the last few days there is no activity being shown for the second node.

Perhaps it's to do with the node's logs not being included in the Operations Monitor for some reason?

Re: How to check if Load Balancing is working for a multi-node Qlik Sense environment?

Giuseppe_Novello — Mon, 29 Apr 2019 12:58:00 GMT

It could be that the Operations monitor is not feeding the correct data which then it will give you the impression nothing is loading to the Second node. In that case I would explore the logs directly from 2nd node and see if you notice anything is being load at all. Logs to check Programdata / Qlik / sense / logs/ engine/ trace : system_engine and Audit_engine.

Gio

Re: How to check if Load Balancing is working for a multi-node Qlik Sense environment?

melissapluke — Sun, 31 Jan 2021 22:10:31 GMT

Hi,

Did you figure out the issue in this case? We are having the same problem.

Thanks,

Melissa

Re: How to check if Load Balancing is working for a multi-node Qlik Sense environment?

mountaindude — Wed, 03 Feb 2021 13:50:00 GMT

I've seen this too on occasion, the root cause usually been firewall ports not being properly configured/opened. Which in turn leads to the central node not being able to hand out incoming requests to the various engine nodes in the Sense cluster. But this would also result in a fair amount of errors and warnings in the log files on the central node.

One way of monitoring this on a real-time basis is to use a tool like Butler SOS (butler-sos.ptarmiganlabs.com).

It's an open source (free!) micro service (created by myself, but used by large and small Sense users around the world) that offers

Close to real-time insight into how many sessions there are on each server in a Sense cluster
What the CPU and RAM usage looks like on those servers
Errors and warnings are highlighted as they happen
Alerts can be configured to notify sys admins within seconds after something going wrong

With this setup you get dashboards like this one.
Here there are three servers and we get one chart for each of them, showing both unique users and number of sessions per server.

While not a solution to your specific problem a tool like this can help you understand how load balancing of incoming requests is done in reality and take action when something isn't working.