16 Replies Latest reply: Aug 16, 2017 4:36 AM by Carlo Zambon RSS

    NPrinting 17.4 Failing and Lots of Tasks running

    Kaitlyn Wright

      Sorry for the long post. We are running 17.4 NPrinting (June 2017) release with Qlikview Server/Desktop 12.10 (SR7) but experiencing lots of issues.  Main issue - server fails and sometimes falls over, about every two days, so reports don't get generated and then a massive backlog of tasks/reports generates.

       

      Stats:

      • QV Server - 2016 - 8 Core - 16GB RAM (QV 12.10 SR7)
      • NPrinting Server - 2012 R2 - 8 Core - 8GB RAM (NPrinting Server 17.4)
      • Apps - 4 Apps, file size between 1MB-2MB each (Broken up to ensure NPrinting has less issues) - reloaded on Qlikview schedule (10minutely - takes less than a minute to reload).
      • NPrinting Connections - 4 Connections one to each App, all QVP Server connections
      • Nprinting Reports - 14 Reports, single Pixel Perfect, no filters
      • Nprinting Task - 1 Task, all 14 reports - output as PNG files only to a folder destination, scheduled to reload on a 5 minute (staged not to coincide with QV) schedule.  Can drop this back to less frequently if required.

      *This same server started life as NPrinting 17.1 server, and has been upgraded through all NPrinting versions.

       

      Concerns/ Questions

      Log files are growing - and iterating into the 10MB every few days - so we have 10 + files of Engine Logs and i found reference to the this- "When a file becomes too big to be easily managed, Qlik NPrinting creates a new file and adds a progressive integer suffix to the old file. So the file without suffix is the current file, and the files with suffixes are the previous files. Bigger suffix indicates older files. The files are not rotated on a daily basis; you have to manually delete old files to save disk space." Is this true, do we need to schedule a clean up task? Could it be that the server is falling over when it gets to the 11th file being 10MB and therefore not knowing how to handle the rename?

       

      When I run the verification on the Cache Generation the apps it fails on the service account check - "The account used to run Qlik NPrinting Engine has access to QlikView Server" however i can confirm - services account running NPrinting - has QV license, is in Admin Group on QV and NPrinting server, also in the QV Admin Group on the QV Server.  So why...? Cache regenerates successfully via task execution status /details.

      eea49f8559.png

       

      The new Task Execution page (firstly i must say is AWESOME!) however, i seem to have issues, after a few hours of running there are about 100 tasks actively still "running" in various stages of % complete....

      a5262e5b3b.png

       

      Occasionally we will get a "warning" in the task, but frequently the tasks complete successfully.  The warnings are all similar they seem to throw something like this (although the objects warning seem to change):

      68ef856099.png

       

      Any help or guidance as to where to start troubleshooting this more is greatly appreciated!

      Thanks!

        • Re: NPrinting 17.4 Failing and Lots of Tasks running
          Lech Miszkiewicz

          Hi Kaitlyn,

           

          Looks like your NPrinting service account is not part of QlikView Administrators Group and Parto of Local Administrators Group on QlikView server??? It also needs to be domain account.

           

          The issue with queued task is that NPrinting is able to start task, but essentialy is not able to fully execute it and finish. This creates a lot of log entries/warnings so your log files are growing fast and slowing down nprinting environment too.

           

          It is just a chain reaction which has an impact on whole setup. I may be able to get back to you tomorrow with more details.

           

          regards

          Lech

            • Re: NPrinting 17.4 Failing and Lots of Tasks running
              Kevin Pinto

              Hi Lech,

               

              Is there a way to limit the number of NPrinting tasks currently executing?  My users have gone crazy with NPrinting and they are hammering the QlikView Server with lots of reports on large documents.  I looked but could not find any place where this could be adjusted. 

                • Re: NPrinting 17.4 Failing and Lots of Tasks running
                  Lech Miszkiewicz

                  There is no way to do this at the moment. Qlik NPrinting is executing all tasks based on amount of CPU cores available on the NPrinting engine box.

                   

                  Are you talking about OnDemand reports? or Users are using NPrinting NewsStand and Subscriptions functionality to trigger huge amount of reports?

                   

                  For big reports on large documents i do not provide OnDemand functionality nor subscriptions as mty major rule!

                   

                  Reporting tools like NPrinting need to managed according to user awareness of impact they are causing. Therefore i never give a full freedom to users or i manage it by user roles and credentials assigned to roles + dedicated reduced documents for NPrinting to take off from QV server some of the cumbersomeness of large documents.

                    • Re: NPrinting 17.4 Failing and Lots of Tasks running
                      Kevin Pinto

                      No, I have a distributed group of Designers and they have created a large number of scheduled reports already, at least some of which are based upon large documents.  We don't use NewsStand or On Demand reports right now.

                       

                      It looks like I will have to monitor NPrinting performance and reach out to the designers to moderate their reports.

                        • Re: NPrinting 17.4 Failing and Lots of Tasks running
                          Lech Miszkiewicz

                          100% - QlikView app design and NPrinting report design go together and require huge knowledge in order to capitalise on its performance. If you have multiple developer who build whatever they want then it is not a right solution. They have to work together in order to:

                          • schedule reports accordingly
                          • do not duplicate similar reports
                          • work with QlikDevelopers (unless they are also Qlik Developers) to maximise QV app speed and responsivenes

                          regards

                          Lech

                    • Re: NPrinting 17.4 Failing and Lots of Tasks running
                      Kaitlyn Wright

                      Hey lech_miszkiewicz

                       

                      Thanks for the thoughts... but this is the odd thing.  We are using a domain account. I'm thinking maybe a full reinstall...? Thoughts?

                       

                      SILVERCHEF\NPrinting is the account -

                      App13 is NPrinting server

                      App 12 is QV Server.  (see screenshots)

                      NPrinting is Admin and QV Admin

                      a17e70a8f1.png

                      f554a08ba9.png

                        • Re: NPrinting 17.4 Failing and Lots of Tasks running
                          Lech Miszkiewicz

                          Hi,

                          You are right that it is there. Did you have QlikVIew server restarted after adding NPrinting user to QlikView administrators and local administrators? Are there any advanced policy rules applied on this server which could over-rule?

                           

                          Can you try do following:

                          • re-intate NPrinting account as a local admin, and qv admin on QV server,
                          • restart QV server
                          • RDP to QV server as NPrinting user
                            • Open QV desktop
                            • Open (locally) QV document you use for NPrinting reports
                            • Save document (here - see if this account has enough right to save and overwrite an existing QV document)

                          This will test whether policies are actually working for you.

                          let me know how it goes.

                           

                          regards

                          Lech

                          • Re: NPrinting 17.4 Failing and Lots of Tasks running
                            Kevin Pinto

                            kaitlynwright,

                             

                            I had the same error that you did, but after adding the NPrinting service account (which was a domain user) to both the local Administrators and the QlikView Administrators group, the error message went away: The account used to run Qlik NPrinting Engine has access to QlikView Server.


                            Perhaps you could try opening an Explorer window on your QlikView server using the NPrinting account?  Or logging into the QlikView server with the NPrinting account?

                             

                              • Re: NPrinting 17.4 Failing and Lots of Tasks running
                                Kaitlyn Wright

                                Hey Kevin, Yeah that all tests fine too... oddd!

                                  • Re: NPrinting 17.4 Failing and Lots of Tasks running
                                    Carlo Zambon

                                    Hi Kaitlyn,

                                    I'm sorry for this issue. Could you please attach or send me a zip file of the NPrinting Logs folder? You can find it under %Programdata%\NPrinting.

                                     

                                    By the way, could you try to:
                                    1) log on in the NPrinting Engine machine with NPrinting service account
                                    2) open the local QlikView Desktop installed
                                    3) Open the qvw document on the QlikView server using QlikView Desktop

                                     

                                    Thanks
                                    Carlo

                                      • Re: NPrinting 17.4 Failing and Lots of Tasks running
                                        Kaitlyn Wright

                                        Hey Carlo

                                         

                                        We've been having lots of this product for a bit now... but happy to try and get it resolved still -  attached are two zips.

                                         

                                        The first (labelled 20170804_VerificationErrorOnly) is a brand new server where the Verification Issue still presents. Despite checking all the updates, taking a new SQL2012 R2 server and complete new install from scratch following all instructions (in help and below) the Verification on a connection still fails.  Note that the cache does generate successfully so it may be that this is a false positive?

                                         

                                        The steps suggested - works fine, can reload from there and it's using a leased license from the server.

                                        By the way, could you try to:

                                        1) log on in the NPrinting Engine machine with NPrinting service account

                                        2) open the local QlikView Desktop installed

                                        3) Open the qvw document on the QlikView server using QlikView Desktop


                                        Second Zip file  (Logsx.zip) - arguably more exciting/detailed is where the numerous tasks were not finishing and failing and the logs were growing over 10mb and over 10 files original server.  I am not sure yet if this still happens on the new server i'm about to configure all the reports and tasks for this.


                                        Thanks in advance for any help!

                                        Kaite

                                          • Re: NPrinting 17.4 Failing and Lots of Tasks running
                                            Carlo Zambon

                                            Hi Kaitlyn,

                                             

                                            I'm sorry for these issues. I will try to answer you below:
                                            1) Regarding the failed verification: this is a false positive, probably related to string case in connection path, e.g., due to a bug writing qvp://PRDwedApp12/NPrinting/MAIN.qvw will give a false positive if the document is actually qvp://prdwedapp12/Nprinting/main.qvw (notice the different case). This has been already solved internally and will be part of NPrinting September 2017 release.

                                             

                                            2) Regarding the logs: they are automatically renamed to .1, .2, etc. where higher numbers corresponds to older log files. The number of file kept depends on the settings included in the following files:
                                            - C:\Program files\NPrintingServer\NPrinting\Engine\Qlik.NPrinting.Engine.config (NP Engine Service)
                                            - C:\Program files\NPrintingServer\NPrinting\Scheduler\Qlik.NPrinting.Scheduler.config (NP SchedulerService)
                                            - C:\Program files\NPrintingServer\NPrinting\Webengine\Qlik.NPrinting.Webengine.config (NP WebEngine Service)

                                             

                                            and in particular to settings like these:

                                            <maxSizeRollBackups value="3" />
                                            <maximumFileSize value="10MB" />


                                            Regarding the Engine service you have probably these settings:

                                            <maxSizeRollBackups value="10" />

                                            <maximumFileSize value="10MB" />

                                             

                                            So 10 files are kept and a new file is created every 10 MB of logs. The default maxSizeRollBackups should be 3 also for the Engine service (like Scheduler one, looking at your log folder) but this should not be an issue (if you have more than 10x10 MB of free disk space, as I suppose).

                                             

                                            3) Regarding the rest of issues I will suggest you to try to run just one task at a time to the new machine you have prepared (on the older one it seems there is a trigger every 3 minutes) and see if it works. In this way we could have focus better on initial errors (that are probably causing other ones).
                                            Anyway I suggest to open a support request so that we can handle it better and solve your issue.
                                            Upcoming September version will be much better than 17.4, so, apart for opening the support case (that I consider a good thing to do anyway), if I was you I will nevertheless give it a try.

                                             

                                            I'm still available to answer your question as soon as possible but for sure we can give you a better support if you contact the Qlik support directly, so that we can assing the best person to the case and if needed arranging a remote session to investigate and solve the issue directly.

                                             

                                            Thanks

                                              • Re: NPrinting 17.4 Failing and Lots of Tasks running
                                                Kaitlyn Wright

                                                Hey Carlo

                                                Thanks for the reply I've been testing a few things out this week (so sorry for the delay).

                                                 

                                                1) on the false positive definitely still getting an error there, have validated case of the text but happy to hear it's resolved in Sep 2017.

                                                 

                                                2) Have played with this a bit and dropped it down to 3 files and only 20k - still issues present (errors in running of schedule - so other than it growing (which is an outcome of the issue, not the cause) understand the rollback options and sizes. Server Disk space is huge so not an issue.

                                                 

                                                3) Okay so yes, we ran tasks successfully for a few days with only two reports being generated (from one Connection) when I add more reports from multiple connections it seems to start "erroring" again.  I have attached the last day's worth of logs.  You see a few things in the logs, one lots of warnings of it not being able to delete the TEMP files. And it gives the "warning" on a connection but only sometimes. Very annoying as it doesn't happen consistently and not only on one document - sometimes it happens on NZ sometimes AU (two different QVWs). It seems that it's keeping a QVW file open which is causing a timeout sometimes....

                                                 

                                                Questions on this then - it seems tasks run sequentially - is there a way to multi-thread this?  If not - should I have a task for each connection? Or should I put a "Fake" task in to ensure the connection gets closed each time... would that work?  Really looking for a workaround now to get this live to users. Any thoughts would help.

                                                 

                                                We have ensured schedules between QV Server and NPrinting don't ever overlap.  There's 14minutes between each - and don't ever overlap the tasks only run for 30seconds when a success but 2minutes when warning/error.

                                                  • Re: NPrinting 17.4 Failing and Lots of Tasks running
                                                    Carlo Zambon

                                                    Hi Kaitlyn,

                                                     

                                                    sorry for not answering you sooner.

                                                     

                                                    Regarding errors

                                                    Since there are lots of warning/errors in the logs and some can be caused by other happened previously it is difficult to completely understand what is happening without having the possibility to test the system directly (and unfortunately this is not possible for us without opening a case to NPrinting support).
                                                    Anyway I'm quite sure that a lot of them have been already solved internally and will be released with upcoming September release (17.5). For example we found and solved a bug regarding concurrent tasks/triggers execution that is probably affecting your installation as well and has been solved in 17.5. Also errors like "Row was updated or deleted by another transaction (or unsaved-value mapping was incorrect)" has been solved in 17.5.
                                                    If you could open a support case we would be glad to check if your system runs correctly with a release candidate of 17.5 and if not to fix the remaining errors.
                                                    Moreover, looking at you execution page and at the percentage of the running tasks I suppose that lots of them are stuck on report genaration phase (I could be sure of it with some screenshots of the table details present within each execution). If so we have improved also that part in 17.5.

                                                     

                                                    Regarding tasks
                                                    Tasks runs in parallel/in a concurrent way on NP 17.
                                                    Briefly:
                                                    - the template of each report is filled with placeholders related to the content we want from QlikView/Qlik Sense (an image, a table, a formula, etc.), so each placeholder is a request of some kind of content;
                                                    - the documents are opened on the NP engines to gather the contents. Only a limited number of document are opened at the same time to avoid slowing the machine down: the number of these connections to QlikView/Qlik Sense depends on your system specs;
                                                    - all the requests regarding opened documents are resolved with the proper tables, images, etc. and then, when all the requests of one document are obtained, the document is built.

                                                     

                                                    Anyway the degree of parallelism strictly depends on your machine specs and your configuration, e.g., how many different connections (path to qlikview documents) you have.

                                                     

                                                    So unfortunately it is difficult to solve the issues you are facing without checking the documents, the templates, etc.: it could be for example that also the templates are not configured in the best way or that QlikView documents are not well structured, and these causes useless delays to get contents and eventually the system to hang. Moreover there could be a misconfiguration with data reload on qlikview server: when a data reload happens the data is considered potentially not updated and so some tasks could be interrupted and then retried due to this: too frequent reloads could cause a situation in which the system is never able to complete current executions.

                                                     

                                                    So I suppose that with recent bugfixes and improvements done in 17.5 and proper support to optimize your configuration we will be able to have your system up and running to being able to fullfill your business needs.
                                                    Please evaluate the possibility to open a support issue so that we could be able to truly help you.

                                                     

                                                    Best regards
                                                    Carlo