Qlik Community

Qlik NPrinting Documents

Documents related to Qlik NPrinting.

Announcements
Customer & Partners, DEC. 9, 11 AM ET: Qlik Product & Strategy Roadmap Session: Data Analytics REGISTER NOW

NPrinting Governance Dashboard

JonnyPoole
Employee
Employee

NPrinting Governance Dashboard

Update June 2021***

NPrinting Governance Dashboard Version2 is out.

To upgrade , simply deploy the new QVF  and reload the app

https://github.com/eapowertools/NPrintingGovernanceDashboard/releases/tag/v2.0

If and only if you have altered the engine.config file(s) on your NPrinting Engine computer(s) to change the default 'resolvers-count' value,  then go to the 'deployment summary' sheet and enter the 'resolvers-count' value under 'Available Resolvers'.  This value sets a baseline which the new concurrency metric can evaluate to ensure your NPrinting environment is not core constrained. 

Please enjoy and post comments / suggestion in github preferably, or in the community comments below.  

This has been tested with NPrinting May 2021 and Qlik Sense May 2021 but it should work with NPrinting April 2020+ .  If you have an old Qlik Sense version it will still load but some of the visuals might not work.  I'm using some newer UI features in Qlik Sense. 

 

Update Summary:

- Updated Deployment Summary Sheet

- Updated Execution Analysis Sheet

- New Report Performance Sheet

- Various UI cleanups and additional filters.  More standardized format

- New configurable variable (vMonthsToLoad) to control how many months of data is loaded from the NPrinting repository.

-New configurable variable (vResolvers) which is set to the number of logical engine processors. 

 

Update Details:

Deployment Summary:

- Configurable variable "vResolvers ".  By default this is set to the # logical engine processors. If you have changed  the "content-resolvers" property in the engine.config file(s)  to increase/decrease the number of resolvers in your NPrinting environment,  set vResolvers to the sum of this value in all your engine.config files.  

-New KPI "Peak  Connection Concurrency".  This is the number of unique connections used at any one time by executing publish tasks, subscriptions, on-demand requests, or metadata reload requests. It does not include preview requests.  If this number exceeds 60% of your resolver count, queuing is expected to occur and reports will take longer to run. The metric will turn red indicating it is time to add logical engine processors to your NPrinting Engine(s) to a maximum of 16 logical processors per engine. 

-New KPI "Peak Execution Concurrency". This is the number of concurrent execution requests at any one time . It is the sum of executing publish tasks + executing subscription requests + executing onDemand Requests + executing Metadata reload requests. If multiple reports are running in a single publish task, it counts as just '1' in this resolver . Does not include 'preview' requests.  

 

Report Delivery:

-New filters added

 

Data Connections:

-New filters added 

-Section Access field (yes/no ) added 

 

Qlik Lineage:

-New filters added

-New KPIs added:  Complex reports , Medium reports, simple reports

 

App Content:

-new filters added

 

Task Recipients:

-new filters added

 

Execution Analysis:

- New time chart:  "Active Connections by Type Over Time".  Shows concurrently used connections over time , stacked bar charts splits the type of request using the connection  (Publish Task, Subscription,OnDemand,Metadata Reload) . A Y Axis limit line shows the peak.  It turns red if it exceeds 60% of the vResolvers variable

-New table "Days when Concurrency Exceeded". Shows the number of days where the peak concurrency connections exceeds 60% of vResolvers.  

- New time chart:  "Executions by Type Over Time".  Shows concurrent executions over time , stacked bar charts splits the type of request   (Publish Task, Subscription,OnDemand,Metadata Reload) . A y axis limit line shows the peak over time.

-New table "Peak Concurrency by Day". Shows peak concurrency by day.

Report Performance:

-Container with 4 distribution plots showing execution length for publish tasks, subscriptions,ondemand requests and metadata reloads.  Colored by status (green = success,  red = failed)

-Container with 4 tables showing detailed executions

- 4 KPIs showing  "Average Publish Task Length", "Average OnDemand Length", "Average Subscription Length", "Average Metadata Reload Length" . 

Users:

-New Filters added

Execution Log Messages:

-New Filters added

 

Load Script / Model:

vResolvers variable set to the sum of logical processors on each NP engine found

- section access field added to connections

- New master date table intervalmatched to executions. 

- vMonthsToLoad variable determines how many trailing months to generate in the master date table (default is 3 months)

 

 

 

 

 

 

 

 

//********************************************************************************************//

Hello, 

I've released the NPrinting Governance Dashboard at the following location. 

https://github.com/eapowertools/NPrintingGovernanceDashboard/releases/tag/v1.0

It is a Qlik Sense App that loads a large amount of information from the NPrinting PostgreSQL repository to help administrators better visualize and understand their NPrinting Deployments. 

summary.JPG

Flow.JPG

 

The app will load and display details about the NPrinting environment whether you use Qlik Sense as the back end or QlikVIew.  But the app itself is a Qlik Sense App. 

recipents.JPG

Requires the use of NPrinting April 2020 and later.  Earlier versions of NPrinting are not supported. 

Sheets include:

1. Deployment Summary

     - Summarizes the version , features deployed and amount of content deployed

2. Report Delivery Overview   

   -   Shows directional report names and volume to desitnations and recipients per month

3.  Data Connections

   -   Over view of all data connections and their status

4.  Qlik Lineage

   -   Shows the lineage from Qlik App -> App Object  -> Report as well as a score of the reports complexity vs executions

5. App Content

    -  Aggregates a list of all tasks, reports, filters etc.. by NPrinting App . Also shows dependencies between all objects (  Tasks -> Reports -> Connections ,   Report Filters -> Filters -> Connections ,  Task Conditions ->  Conditions -> Connections etc...) 

6. Task Recipients

   - Shows a grid of all groups and recipients and which tasks / subscriptions they are configured to receive

7. Execution Analysis

    - Lists all Publish Tasks, Subscriptions, and OnDemand requests showing a range of each execution time

8. Users 

    -  Lists all users and the groups, roles, tasks, subcriptions, ondemand requests that they own or belong to

9. Execution Log Messages 

     -  Shows all messages from the repository about executions.  Does not load information from the logs themselves.

 

Labels (2)
Comments
Ruggero_Piccoli
Employee
Employee

Great job! Thanks!

0 Likes
Lech_Miszkiewicz

Love it!!! Such a great app.

My comments below and possible room for further improvements which are not currently in but (i think) will not be too hard to add:

  • Add Filter details page - this app pulls in information regarding filters giving you info which filter is used and where, but does not expose values/expressions and types (is_advanced_search, is_numeric) of the filters applied. From my experience this is very valuable information allowing you to quickly apply search on "Value" column in "filter_field_value" table to give you full overview of filters which use particular field from Qlik app. I guess this should by quite simple change as it only requires you to add below code to your script + aliases to show user friendly name and/or small transformation to put is_numeric, is_advanced_search, evaluate into single column so its more user friendly... Obviously I can just add it by myself, but throwing an idea on how to enrich this app for all!
    • LOAD id,
      value,
      display_value,
      evaluate,
      is_numeric,
      is_advanced_search,
      filter_field_id,
      If(is_advanced_search='True','Advanced Search',
      If(evaluate='True','Evaluate',
      If(is_numeric='True','Numeric Value','Value'))) as Type,
      upper("filter_field_id") as FilterFieldID;
      SELECT "id",
      "value",
      "display_value",
      "evaluate",
      "is_numeric",
      "is_advanced_search",
      "filter_field_id"
      FROM "public"."filter_field_value";
  • Comment/Question regarding "Complexity markup on Qlik Lineage page: I understand the formula which calculates complexity is quite straight forward. My question is: Are the parameters used in the formula really considered by Qlik as valid when estimating report complexities? I mean all my reports (also those ones which I would consider as extremely simple)  will be complex as quite frankly it is very easy to go over those thresholds. This is just a question out of my curiosity - thats all.

Anyway - thanks very much for such a great effort. I really appreciate this app!!! 

cheers

JonnyPoole
Employee
Employee

hi @Lech_Miszkiewicz   .  Thank you for the feedback.  Yes the field level would be a whole new level of governance and oversight and powerful.  While it would be relatively easy to add the ability to drill into fields for filters,  I was contemplating and ultimately left out (for now) the idea of associating fields to not just filters, but reports, connections, conditions maybe tasks, and ultimately users. That would allow a lot lineage and dependency questions to be answered.   i feel like that is real target to harness the associative magic of the engine but i know it wouldn't be a simple fact table add,  i would need to add new fact type records for most of the fact tables in the model and new link table record tyupes for the link tables.   I've done a lot of that already and the structure is there (i think) to expand the model.  But i'll wait to see what other feedback there is before doing that.  Already the style of the app is one that will likely need updates with every version if/when the repository expands so i'll have to set aside time to keep up with things.  Anyways really appreciate the feedback there.    

For complexity,  i know and agree that most reports will come out as "complex" . To answer your , question its 'non-official' from a qlik standpoint and the rationale has to do with the # of content resolver requests that would be generated by the report and content resolution is the biggest part of report execution which is , basically waiting for NPrinting to issue and retrieve all the data requests from QV or QS   .  I do feel like a report with even a single level that iterates 3-4 objects can result in often dozens of separate associative queries (or content resolver requests). When you take into account all the objects, selection states that are driven by levels, pages, and cycles, it easily becomes a lot of requests. Having 5 objects or a single multiplier (level/page etc..) is enough to bump up the complexity  .  A 4 core  NP engine out won't have enough cores to handle 5 concurrent content resolver requests , and an 8 core engine won't be able to handle  10 requests.  So if you want to run a complex report (10 requests+)  quickly you should have a 12 core engine or engines.    Otherwise ,  and probably in spite of, there will be serial processing most of the time .  So even mildly complex reports can take a bit more time and that's where the designation comes in.  Hope that helps a bit.   if you feel there is another driver of complexity besides this kind of estimation of performance  / load please share .    Once again thank you very much

marcginqo
Partner
Partner

Hi @JonnyPoole,

Really great work, very helpful. Will give it a test drive for sure.

Marc

0 Likes
mikaelsc
Specialist
Specialist

it was written there all along 🙂

mikaelsc_0-1600864969542.png

 

 

Thanks ! 

0 Likes
haty
Contributor III
Contributor III

This is awesome! Thanks Jonny

I got an error (copy below) when reloading. There was no public.smtp_destination_to_to table in my database. I just commented out that part of the code and now works perfectly. We are running Qlik NPrinting February 2020 (Version: 20.0.2.0)

The following error occurred:
Field 'smtp_destination_id' not found
The error occurred here:
SQL SELECT "smtp_destination_id",
"email"
FROM "nprinting"."public"."smtp_destination_to_to"

rzenere_methode
Partner
Partner

Hey @haty ,

I had the same 'issue' on a November 2019, but from my point that should be expected since one of the requirements is having "NPrinting April 2020 and later"
Just as you, I commented that section and everything worked smoothly

Riccardo

JonnyPoole
Employee
Employee

Correct.  In April 2020, new fields were introduced into the repository to support dynamically resolved additional email addresses in the publish task generated recipient emails.   

https://help.qlik.com/en-US/nprinting/September2020/Content/NPrinting/DistributionSchedulesAutomatio...

It's that enhancement that is responsilble for the load error in < APril 2020 versions.    

Commenting out may allow the load script to complete but i'm not able to comment on accuracy of the data .   

0 Likes
Steven_Haught
Creator III
Creator III

Has anyone attempted to build a scheduler off of this data? Is it possible to create a table that shows potential trigger times? This would be the next biggest piece of help in a dashboard such as this! Really great work though, and thank you for the detailed document on how to implement. 

0 Likes
rafalcqlik
Contributor III
Contributor III

Hi @JonnyPoole 

Great application, we have that deployed and is working perfectly.

November 2020 version introduced Audit Trail feature. Are You planning to extend Governance dashboard by adding details from Audit Trail?

0 Likes
Version history
Last update:
‎2021-06-29 02:04 PM
Updated by:
Contributors