Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.
Motivation
Our environment consists of four servers, all accessed via SSO through the same URL:
Central – Manages system administration and authentication.
Scheduler – Handles data reloads and scheduling.
Consumer – Processes and delivers apps to end users.
Development – Dedicated for development and testing purposes.
We encountered two critical challenges with the default load balancing in Qlik Sense:
App duplication errors when publishing
Published apps cannot be duplicated in the workspace environment, creating accessibility and management issues for developers and users.Inefficient use of the development node
The Development node cannot assist in processing when the Consumer node is overloaded. Our goal was to create an intelligent load balancing mechanism that redirects apps to the development node only if the Consumer node exceeds 90% memory usage.
Implemented Solution
To address these challenges, we developed a custom load balancer using:
Flask as an intermediary API to manage balancing rules.
Load Balancing Module Base URI in Qlik Sense Virtual Proxies to dynamically distribute the load based on custom criteria.
Memory monitoring to decide when the development node can be utilized to support traffic.
Balancing Logic
App type verification – If the app is in the development workspace, it must be directed to the proper node.
Consumer node memory monitoring – Allows the development node to assist only if the Consumer node memory usage exceeds 90%.
Dynamic load direction – The balancer selects the best node to open apps, based on available memory and user type.
Code and Configuration
from flask import Flask, request, Response, jsonify # Imports Flask for API handling and JSON processing
import json # Library to work with JSON files
import logging # Logging library for debugging and tracking
import qsAPI # Custom library for Qlik Sense API interaction
import os.path # Module to handle file system paths
from datetime import datetime # Used for time-based logic
from logging.handlers import TimedRotatingFileHandler # For log rotation based on time
# Initialize the Qlik Sense API connection using a proxy and a certificate
qrs = qsAPI.QRS(proxy=‘localhost’, certificate=‘client.pem’)
# Initialize the Flask application
app = Flask(__name__)
# Global variable to store memory usage of the Consumer node
ConsumerMemory = 0
# Logging configuration with daily log rotation
log_handler = TimedRotatingFileHandler(
filename="Logs\\balanceador.log", # Log file location
when="midnight", # Rotate logs at midnight
interval=1, # Rotate logs every 1 day
backupCount=7, # Keep the last 7 log files as backup
encoding="utf-8" # Ensure UTF-8 encoding for the log files
)
# Format the log messages with timestamp, log level, and message content
formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(message)s')
log_handler.setFormatter(formatter)
# Attach the handler to the global logger
logger = logging.getLogger()
logger.setLevel(logging.INFO) # Set logging level to INFO
logger.addHandler(log_handler)
# Function to check the status of the Consumer node based on memory usage and working hours
def CheckUpdateStatus():
global ConsumerMemory # Reference the global ConsumerMemory variable
try:
now = datetime.now() # Get the current date and time
# Check if it is outside working hours (before 8:00 AM or after 7:00 PM) or on weekends
if now.weekday() >= 5 or now.hour < 8 or now.hour >= 19:
return True # Return True to indicate no overload checks are needed
# Check if the consumer memory JSON file exists
if not os.path.isfile('consumer.json'):
return True
# Open and load the JSON file containing memory usage data
d = open('consumer.json')
consumer = json.load(d)
ConsumerMemory = consumer['usedPerc'] # Update the global memory usage variable
# Return True if memory usage exceeds 90%; otherwise, return False
if consumer['usedPerc'] > 90:
return True
else:
return False
except: # Generic exception handling
return True # Assume True (overload) in case of an error
# Function to log access requests with the client IP address
def log_request(client_ip):
logging.info(f"Access: {client_ip")
# Function to extract user information from the "X-Qlik-User" header
def get_qlik_user():
header = request.headers.get("X-Qlik-User", "") # Get the header value
parts = header.split(";") # Split the header into parts
user_info = {}
for part in parts:
if "=" in part: # Parse key-value pairs
key, value = part.strip().split("=")
user_info[key.strip()] = value.strip()
return user_info.get("UserId"), user_info.get("UserDirectory") # Return User ID and Directory
# Function to check if an app belongs to the development environment
def isDev(appid):
if appid == '__hub': # Return False for hub-specific apps
return False
try:
app = qrs.AppGet(appid) # Fetch app details using Qlik Sense API
stream = qrs.StreamGet('full', "@NodeType eq 'Development_Node'") # Fetch streams for development nodes
if app['stream'] == None: # If the app is not linked to a stream
return True
else:
try:
# Check if the app's stream matches a development node's stream
for s in stream:
if s['id'] == app['stream']['id']:
return True
except:
return False # Return False if there's an error
return False
except:
return False
# API endpoint for load balancing logic
@app.route('/balance/loadbalancing/prioritize', methods=["POST"])
def balancer():
global ConsumerMemory # Use the global ConsumerMemory variable
data = request.get_json() # Parse the JSON body of the POST request
app_name = data.get("AppName", "") # Extract the app name from the request data
# Determine if the app belongs to the development environment
isdeveloper = isDev(app_name)
# Check the memory usage status of the Consumer node
consumerOverlay = CheckUpdateStatus()
# Extract user details from headers
user_id, user_dir = get_qlik_user()
# Log the request details
logger.info(f"Request received, App: {app_name} | User: {user_dir}\\{user_id})
# Conditional response based on node type
if isdeveloper: # return only node dev
responde = ({"AppName": app_name, "QlikSenseEngines": [
"wss://qlik.node.dev.com:4747/"
]})
return jsonify(responde), 200
elif consumerOverlay: # return 2 nodes for use round robin
responde = ({"AppName": app_name, "QlikSenseEngines": [
"wss://qlik.node.consumer.com:4747/",
"wss://qlik.node.dev.com:4747/"
]})
return jsonify(responde), 200
else: # return only node consumer
responde = ({"AppName": app_name, "QlikSenseEngines": [
"wss://qlik.node.consumer.com:4747/"
]})
return jsonify(responde), 200
# Start the Flask server on the specified host and port
if __name__ == '__main__':
app.run(host='0.0.0.0', port=8187)
Virtual proxy Configuration
Conclusion
With this custom load balancing script, you can extend the logic to balance apps using any metric, such as:
User-specific rules (based on User ID or Directory).
App type rules (e.g., production or development apps).
Stream-based distribution (depending on the stream assigned to apps).
Resource usage (CPU, latency, or memory as in this example).
Region or location-based balancing for geographically distributed nodes.
This flexibility makes the script highly adaptable to diverse use cases.
