I'm not entirely sure thazt I understand what you are trying to accomplish with these experiments (except for trying to get into trouble). As far as I know, two rules may be reponsible for causing your clusteirng attempts to fail:
- You cannot manage a single service with two QMS, let alone two copies of the same service. Compare this to putting two steering wheels into the same car.
- The ability to cluster depends on the server license. AFAIK you cannot cluster two independently licensed QVS (but I'm not sure about this when using v12.10 or v12.20). Of course this doesn't apply to the DSC which can be clustered at will and without a license for multiple nodes.
Also note that this community is a volunteer effort. If people with required knownledge are unavavailable, you'll have to wait until they are (no promises here). Or if you're in a hurry, you can always contact Qlik Support.
Agreed with Peter, and the error here is quite self descriptive: the license must allow to 2 or more nodes (which you can check in your LEF file in case you didn't purchase the licenses from Qlik yourself, under the key NUMBER_OF_CLUSTER_NODES). Otherwise, only one QVS.exe service is allowed to run simultaneously.
Indeed, in case the same license number is already in use by two or more QlikView Server services in the same network without the proper license, all nodes will go down except for the first which was installed. All these down nodes can be seen in the QMC as "Off Duty" with a yellow mark in the Status tab.
The reason why DSC pointing to the original server does not break is that the license allows you to have one single QlikView Server running, but the services can be spread across several computers, or all in one, the license does not affect to how you install the software but to how many instances you are allowed to have running at the same time.
Similarly for two or more Distribution Services (line in LEF Is NUMBER_OF_XS).
Also as Peter, note that all here in Qlik Community who don't have a star icon along with our names are not Qlik employees and we don't get paid for helping out. Even employees (apart from a couple as far as I know) don't get paid either for contributing here so please be patient and someone, sooner rather than later, will read your question and reply to the best of their knowledge.
Hi Miguel and Peter,
My mistake that I did not mention this but we are using the same license for server X,A and B and the NUMBER_OF_CLUSTER_NODES; allowed by it is 100.
Yet when server Xs QVS is being used in the QVS cluster of server A >> that ERROR comes up.
Also, when server Xs DSC is being used in the DSC cluster of server B >> the ERROR shows up.
But why doesnt the ERROR show up when server As DSC is being used in the DSC cluster of server B ???
and I understand that Qlik Community is a volunteer effort and is a platform which I personally look up to for any kind of Qlik help. It was just that I wanted to reach out to knowledgeable people like you so that this issue can be solved.
Sorry but there is a misunderstanding here. In our case, both publisher (NUMBER_OF_XS;101) and server (NUMBER_OF_CLUSTER_NODES;100) is allowed by both licenses.
Now, there is one server with all the services (with its QMC running) when its DSC or QVS is clustered with another environments DSC or QVS node then the Multiple management services error comes up and stops that particular service.
But, we have such same 2 other environments ( with their respective QMCs running) when its DSC or QVS is clustered amongst them or with any other environments DSC/ QVS - IT RUNS FINE without any error. In that case then why doesnt this Multiple QMC error show up?
FYI : All the 3 environments are configured to use the same service account, hence there is no issue of communication among the services.
Please let me know on this.
Assuming based on the above (correct me if I'm wrong):
- Several clustered environments can be working fine
- Each cluster has its own license with multiple QDS engines and QVS nodes allowed, so for example, there are three different license numbers for each of the three clustered environments
- Only one QMS instance is working in each of the clusters (must be this way, two or more QMS cannot work in the same cluster)
- DSC (Directory Server Connector) can be clustered, but does not require any additional license, only QVS and QDS do need it
- When a DSC server from Cluster1 is set in the QMC of the Cluster2, an error shows up and DSC services go down
- What is the use case for using Cluster1/DSC1 in the QMC of Cluster2 (which already has specified other DSCs, I guess)?
- Which other services go down, if any?
- What do the logs of each service say when set them to highest verbosity in the QMC for each service (network, firewall, permissions, conflicts, file locks, etc.)?
- Are you removing the DSC1 from the Cluster1 before adding it to the Cluster2?
If that assumption is correct, my educated guess is that two different QMS are competing for the same DSC (each service is uniquely identified within the cluster) and this conflict shuts down the services as a precaution.
If my assumption is not correct, please elaborate on how the different computers host each service and what you need to achieve (could well be that all clusters are using the same license, and hence again the competing QMS -each cluster must have its one and only QMS with its own license, several QVS, QDS and DSC can be managed from each QMS).
Side note: be aware that certain configurations in a cluster are synced via .PGO files, which are written, basically, every time a user logs in or times out, and they are stored into the root folder of the cluster as defined in the QMC as well as in the individual QVS nodes in their ProgramData folder (sometimes even in C:\). These individual PGOs are synced between each QVS and the QMS PGO (the one in the root folder).
See I am not only saying this about DSC particularly and I too understand that each service must have a single QMC head.
My point is different, the REAL scenario I am explaining again :
- We have a server named X >> where all services are running including QMS. Thus its a single server installation done. >> 1st environment
- We have another server A >> where all services are installed including QMS. >> 2nd environment
- The QVS of X which already has a QMC head of X >> is used in the QVS cluster of 2nd environment >> Now Servers A's QVS cluster has 2 nodes >> one QVS of A & another QVS of X used for load balancing.
- Thus multiple QMC error is thrown and QVS of X obviously goes down.
NOW the issue :
- As the above error is thrown we are switching off the QMS on server X. Now error is resolved
- BUT, we have another server B >> where all services are installed including QMS. >> 3rd environment
- The QVS of B which already has a QMC head of B >> is used in the QVS cluster of 2nd environment >> Now Servers A's QVS cluster has 3 nodes >> one QVS of A, another QVS of X & QVS of B. (for load balancing purpose)
- BUT, multiple QMC error is NOT thrown for QVS of server B and it stays RUNNING. WHY such?
- All the 3 environments are having the same license applied and both licenses allow
- The above scenario when implemented for experimental purpose on DSC occurs same i.e. X's DSC goes down but not B servers DSC. Why such?
- The logs are not set to high verbosity in our environments.
I understand its bit complex to understand but I hope this time it will be clear.
If I find time, I will draw a diagram so we can discuss over the same, if you already have one, discussing over it would be much less time consuming and clearer.
To remain with Qlik's naming conventions and avoid misunderstandings, you have only 1 cluster, to which you add multiple QVS (or other services) for load balancing. That is OK.
Therefore, you have one single environment, not three.
In this single environment, named CLUSTER, you want to have three different computers with different services each, so the expected output I understand from your experiments is
Cluster name: CLUSTER (name your own if you want to be more specific)
SERVER X (initial server, installation happens here, QMS is also running, it is the server first licensed): QVS Node 1
SERVER A: QVS Node 2 (QMS must be disabled here)
SERVER B: QVS Node 3 (QMS must be disabled here)
You mention that when you want to add the SERVER A to the CLUSTER started in SERVER X, since both have the QMS service running with the same license, services go down, which is expected. Then you disable QMS and services go back to normal, which is expected.
But then you add SERVER B (which supposedly has all services running) to the CLUSTER using the QMC and you see no errors:
Is QMS running in SERVER B?
- If yes
- Does it already have a different license to SERVER X?
- If yes: what for? Then you don't have a cluster, but two clusters (one cluster per serial number). If you want this server to be a member of CLUSTER, remove the server from the cluster, remove the license (leave it with no license at all), unisntall all services you don't want in this computer, and add it back to the cluster using the QMS running in SERVER X
- If not: it uses the same license as SERVER X, but there are two different QMS.exe running in the same cluster --> This is not expected: contact Qlik Support (and engage with Qlik Consulting)
- Does it already have a different license to SERVER X?
- If not, it is expected not to throw any error
When installing a QlikView cluster, you start first with the computer which will host the QMS service, regardless other services which may be also hosted here. When you complete the installation of at least one of the QVS services of the cluster, in the same computer than the QMS or a different one, and in order to have it up and running and test, you add the license for the licenseable component: QlikView Server (QVS). Similarly, once you have installed the first QlikView Distribution Service a.k.a. QlikView Publisher, you license it using the QMC in the original and only QMS running in your cluster. DSC, QVWS and QMS do not need a license to run.
Only after you did that you can add more QVS nodes or QDS engines to the cluster. When doing so, you only install those components you need, for example, only the QlikView Server service, specifying it in the installer. Once every node is installed, you add them to the cluster in the QMC to apply the corresponding license.
You do not install all components in each server that will be part of the cluster, nor you license each separate server which will be part of such cluster, you do it only from the QMC that you installed in the first round.
If you do the other way around, and install separately each computer individually with all components, licensing it and then trying to cluster, then, that message will appear. Why? Because there are at least two QMS services running in different computers with the same license number, and since the QMS cannot be clustered (it's a technical requirement, not a license issue), it understands there is a violation of the license, shutting itself down.
That's why you don't install nor license all servers independently, but only one, and then use the QMC to add more nodes (or remove) to the already existing, licensed cluster.
For this same reason, if you are using virtual machines or instances in Amazon, you just don't clone them after installing and licensing the software, as they will have the same internal SSIDs and will conflict between them. You first have to sysprep them before installing any software, then you license Windows, and only then you should install other software. This could also be a reason why the conflict happens.
Still, if you have more than one QMS service running, working or not, it is an unsupported scenario and it can work, but it can break, which, based on your description, is what is actually happening. Each QMS generates its own set of PGO files which may not be synced and create conflicts or even lockdowns on the files, and disconnections to users, who can be sent to different nodes for the same application during the same session.
This is key to understand how a QlikView cluster works (and very similar to how a Qlik Sense multinode deployment works so far), and to take into account when you scale out, adding more computers to your cluster to balance the load or simply to have more computing power for larger applications.
EDIT: For reference, see the official help for the latest version at https://help.qlik.com/en-US/qlikview/November2017/Subsystems/Server/Content/QlikView-Server/QVSRM_Clustering_Servers.htm
First of all, thanks a lot for describing this in so much detail.
Also, you understood the issue correctly. The issue as you wrote lies in this part :
" But then you add SERVER B (which supposedly has all services running) to the CLUSTER using the QMC and you see no errors:
Is QMS running in SERVER B?
Yes, QMS is running in server B.
Does it already have a different license to SERVER X? No it does not have a separate license.
If yes: what for? Then you don't have a cluster, but two clusters(one cluster per serial number). If you want this server to be a member of CLUSTER, remove the server from the cluster, remove the license (leave it with no license at all), unisntall all services you don't want in this computer, and add it back to the cluster using the QMS running in SERVER X.
If not: it uses the same license as SERVER X, but there are two different QMS.exe running in the same cluster --> This is not expected: contact Qlik Support (and engage with Qlik Consulting) "
So this was the issue, as this is unexpected to happen. But is happening though.
Again, thanks a lot for your support.
Actually we need that QVS os server X node in the CLUSTER for load balancing purpose on all the weekdays when user traffic is high. So during weekdays we are switching off the QMS of server X.
On weekends we need QMS of server X node to trigger some QV tasks, so we are switching it ON again and after task reloads are done we are switching it OFF again.
This is normal and multiple QMC heads must not be there for any service. But, still the abnormal behavior is noticed for Server B's QVS when added to the CLUSTER. Thanks a lot for your help and will raise the same issue with QLIK Support.
Would be very kind of you if you can please look into this below issue as well.
- If yes