11 Replies Latest reply: Jan 31, 2018 5:50 PM by Toni Kautto Branched to a new discussion. RSS

    N Printing Connection with Shared Persistence

    Rohit Kumar

      Hi Techies,

       

      I have configured shared persistence on Sep version and facing performance issue while connecting it to N printing.

       

      I have one CENTRAL, Two Proxies Nodes(Proxy and Engine Services) and Two Scheduler Nodes.

       

      I have created one security rule which SYNC apps to proxy nodes with custom property assigned to Stream which contains apps.

      1..png

       

      Case 1. When I am making connection in N printing with Central Node proxy and executes the N printing Task, It consumes resources of CENTRAL NODE rather than Proxy nodes.

      Case 2. When I am making connection with any of proxy nodes using proxy of Porxy nodes, PERFORMANCE IS VERY BAD as compare two CASE1.

       

      Central node configuration is same as proxy nodes.

       

      My first question is:

      1. Can anyone explain why Qlik is behaving like it, or I am missing something ?

      2. If I SYNC any app to proxy app and open the same app on central node, which node resource it should utilize ?

       

      Thanks in Advance!

        • Re: N Printing Connection with Shared Persistence
          Rohit Kumar

          Please drop comments on concern?

          • Re: N Printing Connection with Shared Persistence
            Rohit Kumar

            mto

            Mike need your help, No is giving any solution on my concern. Please redirect it to right person or if you would help then please do on priority basis.

             

            Thanks,
            Rohit

            • Re: N Printing Connection with Shared Persistence
              Toni Kautto

              Shared Persistence does not synchronize apps to the individual nodes. Each node communicates with the shared persistence storage and repository database as needed.

               

              NPrinting connects to Qlik Sense like any other user. If the user session ends up on the Central node, it means that your configuration load balances the proxy towards Engine service on Central node.

               

              It is not easy to give an answer on exactly what setting you need to alter, as it requires a review of your configuration ans expectation. Have you considered reaching out to Qlik Support for more detailed guidance?

                • Re: N Printing Connection with Shared Persistence
                  Rohit Kumar

                  Please find my comment below:

                  Shared Persistence does not synchronize apps to the individual nodes. Each node communicates with the shared persistence storage and repository database as needed.

                  I have written balancing rule as Node ="Proxy node" and Apps=App1(When I open the app on central node the computername() show central node name and when I open it to proxy it show proxy node) it means then connection of  proxy node and N printing will open the app or I can say utilize the resource of "proxy node" but data will be fetched from central node where my apps and repository stored, but my point is why performance is bad w.r.t connection central node ? Do transferring data from central node to proxy node then open then app on proxy node takes significant time ?

                   

                   

                  NPrinting connects to Qlik Sense like any other user. If the user session ends up on the Central node, it means that your configuration load balances the proxy towards Engine service on Central node.

                  Can you please explain the above comment in detail ?


                  Lets say if I have 30 mb qvf which has IN Memory "30*10"=300 and N printing creates two sessions so multiply this by 2 which will be around 600 RAM utilization, but the same 30 mb qvf is making 60 GB RAM consumption when we executes N printing task.

                  Can you please help me to understand why it is happening?

                   

                   

                  It is not easy to give an answer on exactly what setting you need to alter, as it requires a review of your configuration ans expectation. Have you considered reaching out to Qlik Support for more detailed guidance?

                    • Re: N Printing Connection with Shared Persistence
                      Toni Kautto

                      As I mentioned it is a bit hard to advise on a detailed level as I do not fully see your configuration nor know your business requirements.

                       

                      Without knowing your exact deployment patterns and configuration, my main recommendation is that you review the setup and adapt to the latest options in Qlik Sense. This will give you better resilience and fail-over in case of failure.

                       

                      In Qlik Sense June 2017 and later the typical deployment structure would look like below image.

                      • Two front-end nodes serving users. These nodes are either accessed directly by users, or there is a network laod balancer in front to distribute users over both nodes.
                      • Two back-end scheduler nodes, which also act as active/passive central nodes.
                      • Shared persistence layer in this deployment pattern are placed outside of the Qlik Sense nodes, to allow for Central node fail-over between the back-end nodes.
                      • Repository and storage can reside on same server, or in separate entities. Main requirements are listed in http://help.qlik.com/en-US/sense/November2017/Subsystems/PlanningQlikSenseDeployments/Content/Deployment/Persistence.htm

                      2018-01-29 10_56_47-Qlik Sense Architecture Overview - June 2017.pptx - PowerPoint.png

                      With the deployment pattern established, a user session will always be processed in one of the front-end nodes. Resource will be consumed in the persistence storage layer, as meta data is transferred from repository to the utilized engine and when the app file is copied from storage to the engine node as it is opened.

                       

                      There is no synchronization of information between the nodes. Each node reads and writes to persistence storage as required. If you have poor performance, a likely root cause is that the connection between storage and node does not comply with requirement of <4ms latency and 1Gb bandwidth.

                       

                      If you need guidance on the exact steps on how to reconfigure your current site towards best practice, please reach out to Qlik Support.

                       

                      ----

                       

                      Qlik Sense applications persist on disk on a server, and when a user requests an application it is loaded into RAM if it had not been loaded by a previous user. Multiple users leverage this application in RAM; it is loaded a single time and is not duplicated for multiple users accessing and analyzing it. RAM is freed when the application is no longer used.

                       

                      If you observe multiplied RAM utilization there are other reasons for it, than multiple user sessions. For example consider that user selections genrate calculations, which are cached in RAM. NPrinting typically executed selections quite intensively, as it processes various filters for reports. Increased RAM utilization may simply be a result of cache growing.

                  • Re: N Printing Connection with Shared Persistence
                    Rohit Kumar

                    Hi Toni,

                    First I really want to say thanks for your inputs.

                    Please find my comments below:

                     

                    In Qlik Sense June 2017 and later the typical deployment structure would look like below image.

                    • Two front-end nodes serving users. These nodes are either accessed directly by users, or there is a network laod balancer in front to distribute users over both nodes.

                           Same we have configured!

                    • Two back-end scheduler nodes, which also act as active/passive central nodes.

                           Same we have configured!

                    • Shared persistence layer in this deployment pattern are placed outside of the Qlik Sense nodes, to allow for Central node fail-over between the back-end nodes.

                          We have configured one Proxy node as backup node. I am not sure what exactly this mean, shared persistence layer are placed outside?

                    • Repository and storage can reside on same server, or in separate entities. Main requirements are listed

                            Repository and storage are on central node means on same sever

                    2018-01-29 10_56_47-Qlik Sense Architecture Overview - June 2017.pptx - PowerPoint.png

                    With the deployment pattern established, a user session will always be processed in one of the front-end nodes. Resource will be consumed in the persistence storage layer, as meta data is transferred from repository to the utilized engine and when the app file is copied from storage to the engine node as it is opened.

                     

                    I do agree with you here.

                     

                    There is no synchronization of information between the nodes. Each node reads and writes to persistence storage as required. If you have poor performance, a likely root cause is that the connection between storage and node does not comply with requirement of <4ms latency and 1Gb bandwidth.

                     

                    I will check with networking team to check the latency ?

                     

                    If you need guidance on the exact steps on how to reconfigure your current site towards best practice, please reach out to Qlik Support.

                     

                    ----

                     

                    Qlik Sense applications persist on disk on a server, and when a user requests an application it is loaded into RAM if it had not been loaded by a previous user. Multiple users leverage this application in RAM; it is loaded a single time and is not duplicated for multiple users accessing and analyzing it. RAM is freed when the application is no longer used.

                    I do agree with you.

                     

                    If you observe multiplied RAM utilization there are other reasons for it, than multiple user sessions. For example consider that user selections genrate calculations, which are cached in RAM. NPrinting typically executed selections quite intensively, as it processes various filters for reports. Increased RAM utilization may simply be a result of cache growing.

                    So here, If my app is 30 mb and N printing creates 2 sessions so I assume it occupy 1.2 GB but as you said, N printing do lots of calculation even though its very hard to believe RAM has 60 GB covered because of 30 MB application. It could be two concurrent task running or complexity of project and filters on reports , users.


                    One last point, as I believe, as more RAM and cores are available on Qlik Sense server its good and on N printing 12 cores are enough , more than that creates performance issue. So I am trying to know what should be the best config for shared persistence, I mean how to calculate it.


                    Please correct me if I am wrong or add your input.


                    Best,

                    Rohit

                     

                      • Re: N Printing Connection with Shared Persistence
                        Toni Kautto

                        With multiple back-end/scheduler nodes, you are able to set these up as Central fail over candidates. This means that if the active Central node goes down, a passive central node will take over the central node duties.

                         

                        You can not have the persistence layer (repository database and storage) on the central node or the central node candidates. If the host becomes unavailable, the persistence layer becomes unavailable to all nodes, and your Qlik Sense site becomes unavailable to end-users. For this reason the persistence layer is drawn outside of Qlik Sense nodes in the image above.

                         

                        The persistence layer can be deployed with repository database and storage on different servers, but they can also be deployed ont he same server. The choice depends on your preference and available options in your IT infrastructure.

                         

                        In the image above, there are 4 servers where Qlik Sense services are installed. The persistence layer do not need to have Qlik Sense services running.

                         

                        ---

                         

                        I would suggest the following approach to evaluate memory allocation for your app by manually accessing the app with two different users.

                         

                        1. Add a blank sheet in your app.
                        2. Close the app for all users. Restart Engine service if possible, to clear cache and reset memory utilization.
                        3. Observe memory allocation for Engine process.
                        4. Open app on blank sheet with one user.
                        5. Observe memory allocation for Engine process. The difference from step 3 is the minimum required memory required for the app and one user session.
                        6. Open app on blank sheet with a second user. You should now have 2 concurrent user sessions.
                        7. Observe memory allocation for Engine process. The difference from step 5 reflects the required memory required memory per active user session.

                         

                        Any other memory allocated outside of this is consumed for cached calculations. If you observe a increase for a specific sheet, visualization or selections then this can indicate an incorrect calculation or just room for further optimization. Unexpected memory utilization could for example occur if you aggregate over multiple data islands in your data model or if you have complex and nested IF statements in your expressions.

                         

                        If you observe high memory allocation for NPrinting report generation, the same should happen if you manually replicate the selections applied by NPrinting. This way you can exclude NPrinting as the root cause if the observation, and instead focus on the app in question.

                        For more detailed trouble shooting on the reason for unexpected memory allocation, you need to provide a sample of the app file and steps on how to replicate the memory pattern. If you are not able to share the app file in community, the remaining option is to reach out to Qlik Support for more detailed advice.

                          • Re: N Printing Connection with Shared Persistence
                            Rohit Kumar

                            Thanks Toni for your reply. Your each and every input is very useful.

                            I have read your comments and here are my questions and concerns:

                             

                            1. We do have persistence layer on central node and make one proxy node out of two as fail over candidate but I understand if my central node will go down the fail over candidate won't access anything from my persistence layer(Central). While we are configuring, we were thinking to make a separate persistence layer(where no Qlik service runs) but we had a doubt what if my persistence layer or server where my apps and repository database get down, what is the backup of persistence layer?

                             

                            2. We are primarily using Qlik sense to feed data to N printing, our load balancing is working fine, if we run a task which has 5 reports, so it could generate the report on central node or any of the proxy nodes under one task execution. N Printing reports generation switch here and there between central to proxy nodes on N printing task execution. I believe this is what load balancing is ? If Yes, please tell me how can I stop report generation via N printing on Central node or I can notice that RAM and CPU utilization always more than proxy nodes ?

                             

                            3. When RAM consumption goes to 100%, my N printing task start distributing blank mails without reports and after some time task get failed.

                             

                            Please add your inputs on above three points., I really thanks for your time.

                             

                            Rohit

                              • Re: N Printing Connection with Shared Persistence
                                Toni Kautto

                                My pleasure. Some further thoughts on your questions.

                                 

                                1. Basic backup and restore is as described in Backup and restore Qlik Sense ‒ Qlik Sense. However, depending on your business requirements there are many options and techniques for how you can accomplish resilience for the persistence layer. Just a few examples, as food for thought...
                                  • DFS with replication
                                  • Microsoft Clustering Service (MSCS) for active/passive (SAN based) file system
                                  • VM with recoverability, for example VM vMotion or Hyper V replication
                                  • PostgreSQL streaming failover
                                  • PostgreSQL cluster

                                2. First thing is to confirm that you have setup NP to connect to Qlik Sense through the proxy nodes.Second thing is to review Virtual proxy setting for all front-end nodes, so that you have the desired load balancing. Validate that you only list proxy node engines for proxy nodes virtual proxies, that way they will never utilize any other node for workload balancing. See Load Balancing section in virtual proxy settings Editing a virtual proxy ‒ Qlik Sense.

                                3. RAM consumption should never be 100%. Windows will not be able to manage that in a good way and you will end up getting system wide errors. For this part I would suggest that you reach out to Qlik Support to get help identifying the root cause, and get guidance on mitigating configurations. I will not be able to give you a good enough answer on where your configuration is failing.
                                  • Re: N Printing Connection with Shared Persistence
                                    Rohit Kumar

                                    Hi Toni,

                                    Please find my comments for your latest inputs:

                                     

                                    1. You have defined 5 ways to configure persistence layer, I believe all are there for store repository database and apps out of central node so if if my central node get down, my backup node will start fetching data from that external location(persistence layer).

                                     

                                    2. Yes I have made connection with Proxy nodes but if I remove central node from virtual proxy of proxy nodes, data from Work Section will be disappear. If any user creates a new application it appears in WORK section so if I remove the central virtual proxy from the proxy nodes, these kind of apps will not appear.

                                     

                                    3. Lets say I have made a connection with proxy node and I have two proxy nodes and one central node, both proxy nodes and central nodes have same RAM and CPU, I have noticed consumption of RAM and CPU of central node is more than proxy nodes though reports are generating on both  proxy nodes and central node. My question is, do central node has priority than proxy node ?

                                     

                                    Please add your inputs for above comments.

                                     

                                    Your every input is valuable for me.

                                    Thanks

                                    Rohit 

                                      • Re: N Printing Connection with Shared Persistence
                                        Toni Kautto

                                        All Qlik Sense nodes in a shared persistence deployment access the persistence layer on equal terms. Each nodes retrieves information and files as needed. All changes to data is done directly to persistence layer, there is no copy of data in each node. The only function of the central node in this setup is to coordinate scheduler execution and administrative site tasks.

                                         

                                        Node purpose is one way to affect 'Work' appearance. Review node purpose in your configuration: Editing a node ‒ Qlik Sense

                                         

                                        Your front end proxy node should not load balance over the back-end node(s). The back-end node(s) should only execute app reloads. It is also possible that you have added too much complexity to your configuration, which leads to that you are not able to predict the user session execution. My recommendation on this part is still that you reach out to Qlik Support for more specific guidance based on your configuration, as I can not review your current settings.