31 Replies Latest reply: Dec 8, 2014 3:10 AM by Sijo Joseph RSS

    Clustering Issues

    Julian Kamsan

      Hi,

       

      We recently got a cluster license and after trying to cluster the 2 server we are experiencing inconsistency.

       

      The setup:

       

      - 2 Servers (for the sake of server names lets call server 1 as server A and server 2 as server B)

      - Each server running all the default QV services

      - We have put a VIP (Vitual IP with Stickeness) on top of the 2 server in round robin mode so that the load on the web app is distributed equally

      - The QVD files are located on server A and are shared with a UNC path (\\ servername\C$\path....). Both servers are pointing to a shared drive

      - 64bit server on windows 2008

       

      Issues we are facing:

       

      - Server A where we have put in the cluster license first sees both servers and it appears that the cluster is setup correectly

      - Server B does not see server A. It appears that its pointing to local host even though the cluster license has been put in and all the clustering steps are followed

      - Server A shows the cluster name and Server B shows local host under cluster section as well as access point server connections

      - I have tried to use the QvsStatus tool from server A and see clustering for both server up and running however when going fro B server, I only see localhost

        • Clustering Issues
          Gary Strader

          First problem - you shouldn't install all the default QV services on each server.  In particular, you should only have one QMS installed.  There is supposed to be one master QMS within an environment that coordinates everything between the multiple servers.  If you have multiple the can conflict with each other, and it's possible this is your problem.

           

          Rather than trying to fix it, I would get the official clustering setup document from QlikTech Support and start from scratch.  Otherwise you may spend days troubleshooting a corrupted environment.

            • Clustering Issues
              Julian Kamsan

              Gary,

              Thanks for your inputs. Interestingly enough, I did follow Qlikview white paper documentation on clustering, however it still does not seem to work as expected. It does not say anywhere in the document that all the services should not be installed on the same servers. I think the issue is that the 2nd server does not recognize the 1st server as a cluster and I think there are some settings which need to be configured on the second server to get it recognized.

               

              Have you setup clustering before and if so, did you have to copy or replicate the xml files from master server to the other servers in the cluster?

               

              Thanks.

                • Clustering Issues
                  Gary Strader

                  I have set up clustering many times.  I've never done anything with XML files.  When properly set up, the cluster servers coordinate with the others via the shared PGO files.

                   

                  Can you attach the documentation you're using, and if I have time I will see if it's the same documentation I've used before.

                    • Clustering Issues
                      Bill Britt

                      Gray, I am writing him an email offline right now. I am going to try to explain the differences in setup so he will understand that he doesn't need everything on the second server.

                        • Re: Clustering Issues
                          Mark McCoid

                          Hi,

                           

                          I'm going through the same thing with Clustering.  I'm going to try and attach the document I followed, but would be very appreciative if you have more detailed information.

                           

                          This document (From QV Support) seems focused just on the clustering the server without mention of the other services.

                           

                          Thanks,

                           

                          Mark

                            • Re: Clustering Issues
                              Gary Strader

                              That's the correct document, although there might be a new version out.  The steps in the document aren't very detailed, but the basic idea is that you should only have one QlikView Management Service installed in your environment.  In the minimal possible configuration, you would have three servers.  On the first server you would do a full service install - all five services will be installed - QVS, QVWS, QDS, DSC, and QMS.  On server 2 and 3, you only install QVS.  These two servers are "headless", they don't have a local QMS to manage them.  You would license server 1, and add server 2 and 3 QVS instances to the server 1 cluster configuration in QEMC system/setup.  You do NOT add server 2 and 3 to the "QlikView Servers" list shown in the tree menu in system/setup.  Again, they're headless and you're not supposed to be able to configure them independently of the cluster.

                               

                              As of version 10 you can cluster QDS and DSC, but this isn't something that you would do by default.  The clustering of each service serves a different purpose and it's only worth purchasing and setting it up if you need it.

                                • Re: Clustering Issues
                                  Mark McCoid

                                  Thanks for the clarification Gary.

                                   

                                  Another question...we had originally thought about doing an active/passive failover, but then decided to go with the cluster, however when looking at the recommended configuration, it looks like if the server with the Managment server and web server on it failed, we would be dead in the water.

                                   

                                  Do you have any recomendations on making a cluster more resilant?

                                   

                                  I'm no expert with clusters and the like, so I may have missed something obvious.

                                   

                                  Thanks,


                                  Mark

                                    • Re: Clustering Issues
                                      Gary Strader

                                      In a QlikView Server cluster, the single point of failure are the PGO files which are located on a network share.  Since QlikTech does not currently support storing PGO files on a SAN or NAS, you are stuck with local drive shares.  If the drive fails (hopefully you have RAID), or the server dies or crashes, your cluster is dead.  I am not aware of a workaround or alternative.

                                       

                                      A single web server would also be a single point of failure.  For this reason they are commonly put in a "cluster" of two or more web server machines, with a hardware load balancer directing sticky sessions to them.  You can do this without any special licensing, but it does require a hardware load balancer.

                                       

                                      With any QlikView service, you can always set up a cold failover.  Have servers on standby and when one of the active servers dies, apply the license to the failover machine, repoint DNS, etc.

                                        • Re: Clustering Issues
                                          John Lobaugh

                                          Gary,

                                           

                                          In a QlikView Server cluster, the single point of failure are the PGO files which are located on a network share. 
                                          Since QlikTech does not currently support storing PGO files on a SAN or NAS, you are stuck with local drive shares. 
                                          If the drive fails (hopefully you have RAID), or the server dies or crashes, your cluster is dead.  I am not aware of a workaround or alternative.

                                           

                                           

                                          Could you explain this further? We are running QVS in a HOT HOT configuration with a pair of servers. Don't the pgo files have to be in a shared location accessible by both QVS.exe processes ?

                                            • Re: Clustering Issues
                                              Steve Dark

                                              Hi,

                                               

                                              We have clustering set up and working successfully (QV11 SR1 Cluster Licence).  We currently have two servers clustered and distribution services installed on both servers but the services are stopped on one server (Hot/Cold).  In the event of the server with distribution services failing we can start distribution services on the other server and hook that into QMC and things carry on as before.

                                               

                                              Where we currently have an issue though is if we can not get to QMC because this is on the machine that has failed. In this case we could not bring the distribution services up properly on the other server.

                                               

                                              Our first thought was to install Management Services on the second server so that it can be accessed from either server.  However, the server install documentation (and indeed this thread) warns against doing this.

                                               

                                              Given that, what would the best way forward be?  We are considering a similar approach for QMC as distribution services - where the server is installed on both machines but only running on one at any given time.  Another consideration is not running the management services on either QlikView server - but rather having it on a separate VM that can be imaged and be protected with VM fail over.  This way whatever affects either of the QlikView servers would not also affect the management console.

                                               

                                              Any thoughts anyone has on any of this would be greatly appreciated.

                                               

                                              Steve

                                                • Re: Clustering Issues
                                                  Dave Denscombe

                                                  In that scenario I would install the QMS on two servers but set one to disabled.   Move your QMS application config and QVPR to the share.  You would the need a process in place to ensure only one QMS can be up at any one time.

                                                   

                                                  Looking at other comments in this thread:

                                                  DSC - I always cluster this, there are no cost implications so why not?

                                                  File Share - yes a single point of failure but could be on a Microsoft cluster so will still be a "true" Windows share.  Note it can be a NAS/SAN providing a true Windows share is in front of it (e.g. a VM).

                                                  Web Server - for resilience your should have 2+ of these also with a network load balancer in front to handle incoming traffic.  All web servers would talk to all QVSes.

                                                   

                                                  Thanks

                                                   

                                                  Dave

                                      • Re: Clustering Issues

                                        Hi Bill,


                                        We are currently having one QV server and hosted around 100+ dashboard. Now we are adding one more server and making our environment clustered. We are having the required licences.

                                        When we move to the clustering environment,
                                        should we un-install the existing QV server and install pointing to the SAN storage?
                                        How to retain the .pgo files and QVPR from the existing server.
                                        Should we setup QMC in both the servers ?


                                        Please advise.


                                        Thanks,

                                        Sijo

                                • Re: Clustering Issues
                                  Bill Britt

                                  I would agree with Gary here. This isn't something you can easly fix. I am going to mark Gary's answer as the correct answer also.

                                  • Re: Clustering Issues
                                    Dave Denscombe

                                    In C:\Program Files\QlikView\ManagementService (exact path may differ) there is a file named QVManagementService.exe.config.  Within there is an application base path.  Two caveats though:

                                     

                                    1.) Because the above path is in Program Files it may be overwritten during upgrades

                                    2.) Some of these .config files have some settings you should *never* change.

                                      • Re: Clustering Issues
                                        Steve Dark

                                        Just to make absolutely sure I have this correct, we should:

                                         

                                        1. Stop QV services

                                        2. Find the current folder loction in QVManagementService.exe.config

                                        3. Copy all files from that location to a location on the shared drive

                                        4. Modify QVManagementService.exe.config to point to new location

                                        5. Start up services and test single QMS

                                        6. Stop QMS services on first server

                                        7. Install QMS on second server

                                        8. Modify QVManagementService.exe.config on the second server to point to shared location

                                        9. Start QMS services on second machine and test

                                        10. Test stopping and starting QMS on each each server to ensure switch over solid

                                         

                                        Many thanks for your help.

                                          • Re: Clustering Issues
                                            Dave Denscombe

                                            That's the gist of it yes Steve.  You would need to ensure you were able to be 100% certain that only one QMS can be active at any one time, whether this is done by technology (e.g. MSCS) or having robust manual procedures is up to you.

                                             

                                            The single image solution containing the QMS and maybe even the QDS is more common and I deploy that sometimes too.

                                            • Re: Clustering Issues
                                              Bill Markham

                                              Steve

                                               

                                              I followed your 1 to 10 crib list you posted Sep 6, 2012 11:18 PM in this thread & it worked perfectly - many thanks.

                                               

                                              Best Regards,     Bill

                                          • Re: Clustering Issues

                                            Hello all, I'm 100% new to this whole setup thing.

                                             

                                            I'm trying to follow the instructions in the Clustering White paper mentioned above, but at the most it has 9 bullet points each one sentence long. Either I'm densed or it is simply not giving me enough information. I'm probably densed since most of you seem to understand the document.

                                             

                                            We currently have one production server and after we got that server we purchased a clustering license. So here are the questions.

                                             

                                            1) Install the second server. What are the services that need to run on it?

                                            2) Install the Access Point server. What are the services that need to run on it? There is no mention that I can find about the access point server in the 9 bullet points.

                                            3) Do what on the first server? The picture in the document has two URLs under the Cluster tab. One URL has server 1 and the other has server 2. For some reason server 2 has a port number.

                                             

                                            I had asked the "clustering" question from one of the technical people about setting up the second server and he said that I could fully install the second server and later make the two server a cluster. So that's what I have done until I came here and notices that some servives should not be running.

                                             

                                            Finally, we have a 3rd "test" license. We are installign that on a 4th server. We will be using that of course for testing purposes. According to the technical person I talked to that server at any point can be added to the cluster. Basically the third server can be use to also upgrade to a new version of QV and test everything on it. Once we're satisfied with this new version, we can make that the master in the cluster, upgrade another of the two servers to the new version and make that the second server a node on the cluster. Finally upgrade the 3rd server and make that our test server.

                                             

                                            By his explanation, which I described, it seems to me that a full install is needed on each of the server so they can all be used in a musical chair approach when upgrades are done. Anyone of those server can be the master at any point in time as we move around.

                                             

                                            So I'm basically lost in the clustering maze.

                                             

                                            Thanks for any advice.

                                              • Re: Clustering Issues
                                                Daniel Rozental

                                                Oscar, how many servers are you planning to install?

                                                 

                                                If you have just two servers

                                                Server1: QVS, DSC, QDS, QMS, IIS or QVWS

                                                Server2: QVS, DSC, IIS or QVWS (that's if you don't also have clustered QDS)

                                                 

                                                If you have 3 servers you could do something like

                                                Server1: QDS, QMS

                                                Server2: QVS, DSC, IIS or QVWS

                                                Server3: QVS, DSC, IIS or QVWS

                                                 

                                                If you have 5 servers

                                                Server1: QDS, QMS

                                                Server2: QVS, DSC

                                                Server3: QVS, DSC

                                                Server4:  IIS or QVWS

                                                Server5:  IIS or QVWS

                                                 

                                                When you do an upgrade you must upgrade all services at the same time. QlikView Test server can't be included in the same cluster, you would need to buy another server license for that.

                                                  • Re: Clustering Issues

                                                    Hi Daniel, thanks for the reply.

                                                     

                                                    I have 4 servers:

                                                    • 3 are produciton QV servers
                                                      One of these is the access point
                                                    • 1 server is the test server

                                                     

                                                    In your description which one is the Access Point?

                                                     

                                                    What I was told is to use the 4th server as the test server. When I new version was available, update the 4th server and test. Once satisfied:

                                                    • Take down one of the production servers in the cluster.
                                                    • Change the production license to the test server. So I assume this one will now become the master.
                                                    • Upgrade the production that I had taken down
                                                    • Add the newly upgraded to the cluster
                                                    • And of course, then the last server will become the test by changing its license to the test licese.

                                                    Does this make sense at all?

                                                     

                                                    Oscar

                                                      • Re: Clustering Issues
                                                        Daniel Rozental

                                                        Oscar,

                                                         

                                                        I don't know the specs on your prod servers so I don't know how were you planning to split them, do you have a publisher license in place? I would probably split the QDS and the QMS on their own server and run the AP in both QVS.

                                                         

                                                        I wouldn't do the like that. I would involve moving files and configurations around. I would first upgrade the test server, test everything there, then take down all 3 produciton servers and update them.

                                                         

                                                        There's a very good document about upgrades you should probably read, please

                                                          • Re: Clustering Issues

                                                            Daniel,

                                                             

                                                            Three of the servers (including the test server) are pretty strong (24 cores and 512GB memory). The 4th server, which will be my access point is a small VM since it is just supposed to route traffic. I have a publisher license for all three strong servers.

                                                             

                                                            Doing the upgrade how you mentioned it, is what makes sense to me. But I was just going with I was told by QlikTeck. The reason, of course, if that our reporting is mission ciritical. We don't want to have our reporting down at any time (or at least minimize the down time.)

                                                             

                                                            So does it make sense to setup the environment in produciton to look like:

                                                            Prod1: QVS, DSC, QDS, QMS, QVWS

                                                            Prod2: QVS, DSC, QDS, QVWS

                                                            Access Point: QVWS

                                                             

                                                            Test:QVS, DSC, QDS, QMS, IIS or QVWS

                                                             

                                                            If that's correct, then is there better documentation than the 9 bullet points in the Clustering White Paper that explains the settings?

                                                             

                                                            Thanks again Daniel...

                                                             

                                                            Oscar

                                                              • Re: Clustering Issues
                                                                Daniel Rozental

                                                                Oscar,

                                                                 

                                                                Do you have 1 publisher license for all three environments? then you can only install one QDS.

                                                                 

                                                                Seems correct, sorry but I don't have any other documents on that, you should probably go through the Server reference manual as well.

                                                          • Re: Clustering Issues
                                                            Anusha Vijayakumar

                                                            Hi Daniel,

                                                             

                                                            Our current set-up is we have a SBE server installed on a physical machine. But we are moving to a bigger environment where we have 2 QES clustered in the DMZ, 1 SBE for internal production server, 1 publisher and 1 test server all in cloud. Is there a step by step process on the installation and configuration for my scenario? I understand we have to install different services based on th type of servers. Can you give me in detail which services needs to be installed on each server. Also how would I move all my current production server data to the new production server without any downtime?

                                                             

                                                            Thanks