App Metadata Analyzer (Windows)

[Tyler_Waterfall]

We are excited to share the official release version of the Sense "App Metadata Analyzer" app!

This is a "productized" version of the super app that @Daniel_Pilla and @Levi_Turner posted on Qlik Branch (qs-app-metadata-analyzer).

This app iterates over every application metadata endpoint along with several other QRS calls (Nodes, Apps, Proxies, LB audit), ultimately providing a comprehensive dashboard to analyze your application metadata server-wide. This allows you to have a holistic view of the makeup of all of your Qlik applications, enabling you to have awareness at a granular level of the types of applications in your organization. This application is 100% native to Qlik without any installer, and is easy to configure within the Qlik Sense Enterprise environment as the app takes advantage of the existing 'monitor_apps_REST_app' connection to drive all of the REST calls.

As of the Qlik Sense June 2018 release, a new application level metadata endpoint is available. Data is populated for this endpoint per app post-reload in a June 2018+ environment. You can view this application metadata within your own June 2018+ environment at:

http(s)://<server>/api/v1/apps/<GUID>/data/metadata

where <server> is your Qlik Sense Enterprise server and <GUID> is the application ID. Note that the application is not lifted into RAM as metadata is fetched.

Data from this endpoint is derived as part of the app reload process, and therefore does not include any object or expression related metadata. The data from the endpoint includes:

* server metadata including number of server cores, total server RAM

* reload time

* app RAM base footprint

* field metadata including cardinality, tags, total count, RAM size, synthetic keys

* table metadata including fields, rows, key fields, RAM size, circular references

For a deeper analysis of the application including a demo video, please refer to: https://adminplaybook.qlik-poc.com/docs/asset_management/apps/analyze_app_metadata_analyzer.html

June 2020 Update: In version 2.2.1 and forward of the App Metadata Analyzer, a new Alerting sheet has been added, along with two new variables in the load script. The purpose of this sheet and added capability is to make integration with Qlik Alerting as simple as possible. This new capability and view allows for a Qlik administrator to easily see what applications have breached what thresholds (as well as how many) and be quickly alerted on them. It also allows for the administrator to disable alerts for specific applications and mark others as under review, which might have a different cadence. Please see the attached guide for much more detail and example policies/configurations.

Version History

• v2.2.2 (July 2020) >
  • Replaced the "AppName" field with the "AppNameUnique" field in many visualization objects to properly visualize identically named apps.
• v2.2.1 (June 2020) >
  • Resolved incremental load issue to include imported apps with metadata that had not yet been reloaded on the server
  • Resolved issue with application tag values not appearing
  • Resolved autonumber collision issue with "FieldNameUnique" and "TableNameUnique" fields, due to AutoNumber() being used pre and post QVD
  • Resolved issue where some threshold breaches were stored to QVD, forcing applications to be reloaded before a change of threshold would take effect
  • Removed system fields and system tables
  • Modified chart "Field # Records" to "Field Cardinality" on "Dashboard" sheet
  • Modified "Threshold Analysis" sheet to use boolean logic for measures over/under thresholds vs a dynamic range
  • Added in the sheet "Alerting" for visualizing threshold breaches and allowing for simple tie-ins to Qlik
  • Alerting and NPrinting
  • Added flagging of circular references
  • Added ability to ignore applications from "Alerting" page with specific tag value
  • Added threshold reference lines to "Dashboard" objects
• v1.1.0 > Fixed Incremental loading bug (field count changes over time) (QB-981)
• v1.0.2 > Added back error handling "loop" in load script to handle missing apps (QLIK-98225)
• v1.0.1 > Small fix to correct the KPI for Table Footprint
• v1.0.0 GA > First official release, compatible with June 2018 Sense and later, target release September 2019
• Required Configuration: In the "configuration" section of load script, update the vu_central_node_host_name and vu_virtual_proxy_prefix variables as described in the load script

Sheets:

Dashboard

Threshold Analysis

App Analysis

Alerting

App Availability

[rzenere_avvale]

This seems great! I'll give it a try in the next weeks!

[s_kabir_rab]

Great stuff. Thank you.

[rzenere_avvale]

Hi @Tyler_Waterfall ,

Is there a way to exclude the empty applications (the ones that have been just created or opened without  data)?
In those cases the endpoint returns nothing, and the reloads fails.

I wouldn't like to use a where clause with a fileSize >= (value) since even if empty, the application could have different sizes (i.e. if I upload an attached content)

Many thanks,
Riccardo

[Tyler_Waterfall]

Good question @rzenere_avvale . So you want to exclude apps which have no data (no rows) or no size on disk (fileSize) or?

What about apps used to generate QVDs which drop data at the end?

[rzenere_avvale]

Apps that generate QVDs... Depends. Sometimes we leave some statistics fields, so those have rows.
But definitely when the app has no rows/no size on the disk it would be nice to exclude it!

I thought again about the apps with attached files but have no rows, and those cases should already be excluded since the fileSize is the one of the app without attachments.
Unluckily I believe to have an issue with some application that have data and have been reloaded in September 2018, but I need to investigate more in order to understand why.

Taking a look at the response of the http(s)://{server}/api/v1/apps/{GUID}/data/metadata I see there are always (or most of the times?) three $$SysTable. Do you have any idea about what those are?

Riccardo

[victorbertoldo]

Hi Tyler!!! Great stuff man!

But I did't make it. 

Can you help me?  Thanks.

Victor Bertoldo.

[Tyler_Waterfall]

Try the 1.0.2 version of the App Metadata Analyzer I just published today. @victorbertoldo 

[victorbertoldo]

Where is it? I did't find on github.

[Tyler_Waterfall]

@victorbertoldo  - Find the app here in the Qlik Monitoring & Management Topic Group under Documents > App Metadata Analyzer