App Metadata Analyzer (Windows)

[Tyler_Waterfall]

We are excited to share the official release version of the Sense "App Metadata Analyzer" app!

This is a "productized" version of the super app that @Daniel_Pilla and @Levi_Turner posted on Qlik Branch (qs-app-metadata-analyzer).

This app iterates over every application metadata endpoint along with several other QRS calls (Nodes, Apps, Proxies, LB audit), ultimately providing a comprehensive dashboard to analyze your application metadata server-wide. This allows you to have a holistic view of the makeup of all of your Qlik applications, enabling you to have awareness at a granular level of the types of applications in your organization. This application is 100% native to Qlik without any installer, and is easy to configure within the Qlik Sense Enterprise environment as the app takes advantage of the existing 'monitor_apps_REST_app' connection to drive all of the REST calls.

As of the Qlik Sense June 2018 release, a new application level metadata endpoint is available. Data is populated for this endpoint per app post-reload in a June 2018+ environment. You can view this application metadata within your own June 2018+ environment at:

http(s)://<server>/api/v1/apps/<GUID>/data/metadata

where <server> is your Qlik Sense Enterprise server and <GUID> is the application ID. Note that the application is not lifted into RAM as metadata is fetched.

Data from this endpoint is derived as part of the app reload process, and therefore does not include any object or expression related metadata. The data from the endpoint includes:

* server metadata including number of server cores, total server RAM

* reload time

* app RAM base footprint

* field metadata including cardinality, tags, total count, RAM size, synthetic keys

* table metadata including fields, rows, key fields, RAM size, circular references

For a deeper analysis of the application including a demo video, please refer to: https://adminplaybook.qlik-poc.com/docs/asset_management/apps/analyze_app_metadata_analyzer.html

June 2020 Update: In version 2.2.1 and forward of the App Metadata Analyzer, a new Alerting sheet has been added, along with two new variables in the load script. The purpose of this sheet and added capability is to make integration with Qlik Alerting as simple as possible. This new capability and view allows for a Qlik administrator to easily see what applications have breached what thresholds (as well as how many) and be quickly alerted on them. It also allows for the administrator to disable alerts for specific applications and mark others as under review, which might have a different cadence. Please see the attached guide for much more detail and example policies/configurations.

Version History

• v2.2.2 (July 2020) >
  • Replaced the "AppName" field with the "AppNameUnique" field in many visualization objects to properly visualize identically named apps.
• v2.2.1 (June 2020) >
  • Resolved incremental load issue to include imported apps with metadata that had not yet been reloaded on the server
  • Resolved issue with application tag values not appearing
  • Resolved autonumber collision issue with "FieldNameUnique" and "TableNameUnique" fields, due to AutoNumber() being used pre and post QVD
  • Resolved issue where some threshold breaches were stored to QVD, forcing applications to be reloaded before a change of threshold would take effect
  • Removed system fields and system tables
  • Modified chart "Field # Records" to "Field Cardinality" on "Dashboard" sheet
  • Modified "Threshold Analysis" sheet to use boolean logic for measures over/under thresholds vs a dynamic range
  • Added in the sheet "Alerting" for visualizing threshold breaches and allowing for simple tie-ins to Qlik
  • Alerting and NPrinting
  • Added flagging of circular references
  • Added ability to ignore applications from "Alerting" page with specific tag value
  • Added threshold reference lines to "Dashboard" objects
• v1.1.0 > Fixed Incremental loading bug (field count changes over time) (QB-981)
• v1.0.2 > Added back error handling "loop" in load script to handle missing apps (QLIK-98225)
• v1.0.1 > Small fix to correct the KPI for Table Footprint
• v1.0.0 GA > First official release, compatible with June 2018 Sense and later, target release September 2019
• Required Configuration: In the "configuration" section of load script, update the vu_central_node_host_name and vu_virtual_proxy_prefix variables as described in the load script

Sheets:

Dashboard

Threshold Analysis

App Analysis

Alerting

App Availability

[AliAzizi]

Hi Tyler,

On our customer's Qlik Sense platform we have disabled Centralized logging. Since this change for a couple of months ago the Metadata Analyzer stopped working as the QLog connector is removed from the platform. 

So far we have not been able to find out whether we can get this app to utilize the physical logfiles and QVD files to reload the log data, like the other monitoring apps which are working as intended (e.g. Operations monitor).

Do you know any way to get this Metadata app to work again in an environment where centralized logging is disabled? Or the new version of app can do so?

Best regards

Ali

[Levi_Turner]

@AliAzizi : This app does not use Centralized Logging in any way whatsoever. What is "not working"? Is there an exception in the script log?

[AliAzizi]

Sorry, my bad. It is not this app but Sense System Performance Analyzer which is not reloading after the centralized logging was disabled.

Sorry for this mistake.

Br

Ali

[Levi_Turner]

@AliAzizi : Ahh. The Sense System Performance Analyzer exclusively uses Centralized Logging due to exclusively present information (i.e. performance counter information for services other than the Engine) which is not found in the underlying log files. So it will just not work if Centralized Logging is not configured / running / working.

[AliAzizi]

Thank you for helping.

Br

Ali

[crichter14]

This looks amazing - but I have a question - can it drill down to sheet owners?  In the case that we have multiple contributors, we'd like to see who 'owns' each sheet so that they can be contacted should the sheet be too larger (taking up too much load time) etc.  Thank you!

[Levi_Turner]

@crichter14 : What type of analysis are you after? I ask because this app provides insight into the data model of a Qlik app. It does not do any interrogation of what elements of the model are used (or unused) in any sheet or visualization inside of the sheet.

[crichter14]

Hi Levi!  I might be barking up the wrong tree - but at times we make data changes and we want to know who is using the change (so let's say a field changes its base underlying assumptions).  We want to know how to reach out to a contributor who used that field or created that sheet so we can make sure they know there was a change.  Does that make sense?  Is that anything I can use to do that?  I was using the Document Analyzer but it doesn't include users/owners.