Using Rules to segregate Consumption from Development

[Levi_Turner]

Goal:

The goal for this guide will be to use the rules capability in Qlik Sense Enterprise on Windows to segregate consumption of published Qlik apps (production activities) from ad-hoc development in the Hub. This guide will use both Load Balancing Rules in the first example. In the second example, we will use Security Rules in order "hide" the Streams when accessing the Development Virtual Proxy.

Example 1: Basic Segregation

For this example, we will review basic segregation of Qlik apps across multiple virtual proxies. In order to be successful with basic segregation the Engine(s) which will host the Qlik apps need to not include the Central Node. This is due to some un-modifiable rules in Qlik Sense Enterprise on Windows which will interfere with things [1].

To achieve basic segregation, the administrator will need to:

• Create a Custom Property named NodeType, which is applied to Nodes and has the possible values of Production and Development.

• Apply the Production value to the production only node(s) and ensure that the Node purpose for that node(s) is set to "Production" [2]

• Apply the Development value to the ad-hoc development only node(s) and ensure that the Node purpose for that node(s) is set to "Both" or "Development".

• Disable the default Load Balancing Rule ResourcesOnNonCentralNodes

• Create a new Load Balancing Rule.
  • Name: Separate Production from Development
  • Resource Filter: App_*
  • Actions: Load balancing
  • Conditions: ((node.@NodeType="Production" and !resource.stream.Empty()) or (node.@NodeType = "Development" and resource.stream.Empty()))
  • Context: Both in Hub and QMC

• Create a virtual proxy for the consumption of production apps. In this example it will have a prefix of prod, but this is not required. Use of the default virtual proxy is possible for this. Select the Server Node which is a production node.
  • Not shown: Configure any needed host white list values (Virtual Proxy > Edit > Advanced > Host white list)

• Attach to the desired Proxy (if needed)

• Create a virtual proxy for ad-hoc development. In this example it will have a prefix of dev, but this is not required. Use of the default virtual proxy is possible for this. Select the Server Node which is a development node.
  • Not shown: Configure any needed host white list values (Virtual Proxy > Edit > Advanced > Host white list)
• Attach to the desired Proxy 

• Open up the Hub on each virtual proxy to validate
• Dev:

• Production:

Limitations of Example 1

This approach does isolate the availability of apps between Qlik Engine nodes. Since there is no inherent relationship between the virtual proxy which the user used and the evaluation of the streams that the user has access to, then this approach will result in empty streams when accessing Qlik Sense Enterprise using the dev virtual proxy:

For most organizations this is an acceptable solution. For those who want to further customize the experience, if they are using a supported authentication mechanism (e.g. SAML), then they can implement Example 2 (see below).

Example 2: Advanced Segregation

Building off of the configuration done in Example 1 (above), we can further customize the experience by "hiding" the streams when the user accesses Qlik Sense Enterprise using the dev virtual proxy. This approach uses SAML for authentication due to the ability to pass static attributes on a per virtual proxy basis. Not covered but this approach can be adapted to any authentication type which allows the passing of session attributes. This includes JWT and Web Ticketing. Both of those mechanisms can have customized authentication modules which pass a session attribute which specifies a value which can be further used in the site's security rules.

To implement this approach:

• Edit the Dev virtual proxy and specify a static attribute to be passed like so:

• Customize all security rules which provide stream access to include the clause: and user.environment.access.Empty()[3] 
  • Note: In this example, I am have only a single custom security rule which grants access to my custom streams based on matching a user's group from a User Directory Connector to a custom property assigned to the stream. In your environment, you may have many rules which apply to streams which need adjustment.
  • Note 2: In this example I have likewise adjusted the default security rules StreamMonitoringAppsRead (which governs access to the Monitoring Apps stream) and StreamEveryone (which provides access to the Everyone stream)

• Open the dev Virtual Proxy to validate:

Extending Example 2

This guide shows how to segregate consumption from development using static attributes on a SAML virtual proxy. A similar approach could be done which applies different values for the Access static attribute on each virtual proxy. This would allow segregating both apps and streams:

• across business units 
• between internal and external access

[1]  These rules are the ResourcesOnCentralNode Load Balancing Rule and the OwnerRead Security Rule. ResourcesOnCentralNode ensures that all applications are load balanced to the Central node's Engine. This rule cannot be modified due to architectural assumptions in the design of Qlik Sense Enterprise. For example that apps which need to be migrated will be migrated by the Central node's Engine. The OwnerRead security rule ensures that the owner of some entity (e.g. app, stream, extension, sheet, etc) can see that entity. ResourcesOnCentralNode's consequence is that if the Central node's Engine is used for load balancing then all applications will be available to the user in the Hub. OwnerRead's consequence is that the owners of unpublished apps will be able to see those apps if the Central node's Engine is used for load balancing.

[2]  The Node Purpose value of Production disables the Create App button in the Hub over and above any security rules or license assignments (i.e. a Professional license) which would otherwise grant the ability to create an app.

[3]  Also ensure that all streams are owned by an internal account. As with note [1] above, if a user is the owner of a stream then they will see that stream despite the configuration in Example 2.

[RenzoH]

Thank you Levi!!

[shaybitton]

Very useful post,

Thanks!

[pablodelcerro]

Hi Levi, great post.

In this example, you have 3 nodes, right? Central, RIM Prod and RIM Dev.

The VP are not balancing the Central node, right? So no user conssumption or development use central node resources, right?

But Central node will execute reload tasks? 

Also, VP are linked to Central Node Proxy? right? How will you estimate Central Node requirements?

Another question regarding Example 2, it can also be done with AD Groups, right?

Thanks!

Pablo

[Levi_Turner]

> In this example, you have 3 nodes, right? Central, RIM Prod and RIM Dev.
> The VP are not balancing the Central node, right? So no user conssumption or development use central node resources, right?

Correct. The rationale for this choice is that the Central Node has all applications load balanced to it (driven by the default rule ResourcesOnCentralNode).

> But Central node will execute reload tasks? 

In my lab environment, yes. But the Central node will only execute reload tasks if it is configured to be a Manager and Worker (old verbiage: Master / Slave). If you had a large environment where you wanted to isolate work-loads off of the Central then you'd configure it's scheduler to be a Manager and it would not perform reload tasks.

> Also, VP are linked to Central Node Proxy? right? How will you estimate Central Node requirements?

In this guide I did use the Central's Proxy to perform authentication but the actual application delivery was via the RIMs. This too is configurable. In a production environment, I'd expect the configuration to use one or both of the RIM node's proxy service. As far as sizing, it'd really depend on the work-loads. To use my simple lab environment where I had a Central (performing reloads) + 2 RIMs (delivering apps), I'd size based on the reload requirements for the site. I hedge here because we have customers running production work-loads using 4 core / 32GB servers, and we have others with 64+ cores / 1TB of RAM. It varies quite widely.

> Another question regarding Example 2, it can also be done with AD Groups, right?

Yes. So example 2 is a bit of a trick. We are using a static attribute in a SAML virtual proxy (meaning it has a fixed value which is _not_ coming from the SAML provider) to complement the user's groups from the configured User Directory Connector. I used Active Directory in my lab, but the UDC can be any supported source. One note about example 2 and I don't have a lab setup to test it out but I suspect the need for Example 2 is gone with the release of the Empty streams do not display in the hub feature present in the May 2022 release of Qlik Sense Enterprise Client Managed.

[pablodelcerro]

Levi, thanks a lot for your quick answer!!

I have some more questions, i´m sorry to bother you:

- regarding sizing, my doubt is how to size a Central Node that acts as Manager and is not balanced on any VP so it has no user consumption? A minimum 4 cores 12GB Ram?

- To separate users consuption and reload tasks, like it was before with Server and Publisher, we have to separate scheduler from engine? Load balancing works the same way for both? How do we filter the load balancing rule to work only for engine or scheduler?

- Is there any way to configure schedulers not to run tasks on working hours? On QV we have the Graceful Shutdown.

Again, thank you very much for your help!

[Levi_Turner]

> regarding sizing, my doubt is how to size a Central Node that acts as Manager and is not balanced on any VP so it has no user consumption? A minimum 4 cores 12GB Ram?

If it truly is just hosting PostgreSQL, then that's probably fine. Though from an architectural standpoint most folks from Qlik tend to recommend using the Central as a worker scheduler. Otherwise that's 4 cores / 12GB of RAM doing nothing but back-end work. It's a bit of a waste. There are some narrow architectures where this would make sense, but by in large, it's not preferred. It does add some resiliency should folks reload the wrong set of apps as tasks at the same time, but at a cost of some compute not optimally being used.

> To separate users consuption and reload tasks, like it was before with Server and Publisher, we have to separate scheduler from engine? Load balancing works the same way for both? How do we filter the load balancing rule to work only for engine or scheduler?

If we're using an analogy to QlikView architectures:

• Scheduler ---> Distribution Service / Publisher
• Engine --> Combination of QVS and the QVB processes spawned by Publisher

The Worker Scheduler talks to its _local_ Engine (much like Publisher --> QVBs). As for load balancing, the rules can apply to either / both tasks or consumption. You enable tasks by activating the scheduler service. You enable consumption by linking to a virtual proxy. If you do both, then the node does both. If you do just 1, then the node will just do what's configured. In both cases, you can pin apps to those nodes which will be honored on the configured app usage (e.g. a reload node with pinning will only reload the specified apps, a consumption node with pinning will only open the specified apps).

> Is there any way to configure schedulers not to run tasks on working hours? On QV we have the Graceful Shutdown.

Unfortunately there isn't a clean answer here. I haven't explored this area heavily but it may be possible to use the REST APIs to de-activate the Scheduler service on a schedule.

[pablodelcerro]

Levi, thanks a lot for your answer...

Now im thinking if its better Central node as Manager as I thought before, or as worker as you recomend.. my concern is that happens with heavy tasks being executed on the central node and using a lot of resource on working hours. IF that wont affect the overall performance of the RIM nodes. Any way to limit the amount of cores being used by the scheduler?