The Reload Analyzer for Qlik Cloud Customers

[Jamie_Gregory]

• “How can I monitor reloads across the tenant?” 
• “How can I see what data connections and files are being used?” 
•  “How can I view reload concurrency and peak reload RAM over time?” 

To help answer these questions, we are happy to share with you the capabilities of our Reload Analyzer for Qlik Sense SaaS! 

The Reload Analyzer app provides insights on: 

• Number of reloads by type (Scheduled, Hub, In App, API) and by user 
• Data connections and used files of each app’s most recent reload 
• Reload concurrency and peak reload RAM 
• Reload tasks and their respective statuses 
• And much more! 

(Available sheets) 

The Reload Analyzer uses Qlik’s RESTful APIs to fetch all the required data and stores the history in QVD files, allowing for efficient reloads and historical analysis. 

A few things to note: 

• This app is provided as-is and is not supported by Qlik Support. 
• It is recommended to always use the latest app. 
• Information is not collected by Qlik when using this app. 

The app as well as the configuration guide are available via GitHub, linked below. 

• QVF: https://github.com/qlik-oss/qlik-cloud-reload-analyzer/releases/latest/download/reload-analyzer.qvf
• Release Notes: https://github.com/qlik-oss/qlik-cloud-reload-analyzer/releases/latest
• Installation Guide: https://github.com/qlik-oss/qlik-cloud-monitoring-apps/releases/latest/download/qlik-cloud-monitorin... 

Any issues or enhancement requests should be opened on the Issues page within the app’s GitHub repository.

Be sure to subscribe to the Qlik Support Updates Blog by clicking the green Subscribe button to stay up-to-date with the latest Qlik Support announcements. Please give this post a like if you found it helpful! 

Kind regards, 

Qlik Digital Support Team 

Additional Resources:

Our other monitoring apps for Qlik Cloud can be found below.

• App Analyzer
• Entitlement Analyzer
• Access Evaluator
• OEM Dashboard (for OEM Partners and multi-cloud tenants)

[dmwilliams888]

We're trying to use this in our tenant as fills a big gap in our monitoring but experiencing failures due to the number of responses being too large. This results in not being able to refresh the app.

[AlexOmetis]

Hi, similar to @dmwilliams888 we're seeing a couple of errors across at least 3 of the SaaS environments we have access to:

1. Internal Server Error: (Connector error: HTTP protocol error 500 (Internal Server Error): {"error":"query database"})
2. Message too large: (Connector error: Received message larger than max (8840029 vs. 4194304))

We have only been able to load 1 day of data in one of the busier environments, otherwise we see one of these errors every reload. 

Are there known issues with the APIs this uses that are going to be resolved so we can reload this app (and others like it that encounter similar issues)? Or is there a workaround (other than using ErrorMode=0 and missing any reloads that give this error)? 

[Daniel_Pilla]

Hi @AlexOmetis & @dmwilliams888,

1. Internal Server Error: (Connector error: HTTP protocol error 500 (Internal Server Error): {"error":"query database"})
   1. This is documented in the configuration guide as typically occurring when paginating across many, many pages. In one of our tenants internally (that has hundreds of thousands of reloads), we are not able to load the full 90 days by default, and have to start around a couple of weeks and then build forward incrementally. If you are hitting this issue, I encourage you to open a case with support, as the API is public and stable. I would open it in the context of the API not working (you can paste the URL that errored into the browser and you should be able to re-create it outside of the app that way by seeing the error there), not the Reload Analyzer, as the API is supported by Qlik while the app directly is not.
   2. I would encourage for the time being setting the vu_initial_days_back variable on the *OPTIONAL CONFIG* tab to something less than 90, and then continue to build incrementally from there.
2. Message too large: (Connector error: Received message larger than max (8840029 vs. 4194304))
   1. *EDIT* By default, the app loops over the reloads endpoint with a page size of 100. If you are receiving this error, its likely that some of the responses in the body are very large, and the total of 100 responses breaches that hard response size limit of 4 MB for the REST connector. A new variable called vu_reloads_page_size found on the *OPTIONAL CONFIG* tab  has been added so you can control the amount of reloads that are brought back per request. This can be lowered to something lower than 100 (down to 1) to reduce this response size if needed. If a single reload is too large, you may have to add `SET ErrorMode = 0` at the top of the "Main" tab of the script to roll over the reloads that are too large (you'd want to set it to `1` in that case, though the reload time would be significantly longer).

Thanks for the feedback, and I hope this helps.

Cheers,

[AlexOmetis]

@Daniel_Pilla - really appreciate the detailed feedback!

Yes, we had used the vu_initial_days_back to reduce the data set and managed to get it to load for 1 day in one environment but 30 in another and 90 in another. We'll look into logging the issue through a support ticket.

As for the message too large - I've encountered this before and have found that, in some case, even setting the limit to 1 doesn't get around this - the API is returning a message too big even for one reload! There's no option in the REST connector to increase the maximum size as far as I know and you can't reduce the size by modifying the query sent to the endpoint (other than the limit) as far as I can tell.  I'll see if I can tweak this app successfully... 

[Daniel_Pilla]

@AlexOmetis - I've updated a new version of the app that first and foremost fixes an incremental load bug (sorry about that!) as well as adds a new variable to control the page size for audits. Refer to the changelog above. As far as the REST body size being too big even with a single reload, if you have a very large load script of an app, the REST connector appears to have a 4 MB response size that is hardcoded (referring to the bytes above). I'll dig in with R&D to see what if any our options are here. For the time, you'll have to push through it with `SET ErrorMode = 0`. So you'll want to ensure you delete the QVDs and grab the new app for the bug fixes as well as the new variable addition.

Cheers,

[Kevin-M]

Hi,

This app looks to be more comprehensive than Operations Monitor App, can this be reconfigured for on-premise Qlik Sense installations for customers not using SAAS?

Thanks,

Kevin

[Daniel_Pilla]

Hi @Kevin-M,

Qlik Sense Enterprise on Windows has a very comprehensive application specific to reloads that ships with the product called the Reloads Monitor. The Operations Monitor does in fact include reload data, however it is intended to give you an overall view of the entire site, so it doesn't dive in as deeply. The Reloads Monitor is one of a handful of additional applications that are purpose-built for a specific subject areas. You can find all of the supported monitoring apps for QSEoW here.

Cheers,

[AlexOmetis]

@Daniel_Pilla - it's actually the reloads endpoint (not audits) that I'm seeing giving us the message too big, so I've changed this line in sub get_reloads to make that page by the same number as the audits:

SET vParams = 'limit=$(vu_audit_page_size)';

I still hit both errors in different environments so am now trying with ErrorMode=0 to get round the problematic ones.

Update 1: I have managed to get it to run in one environment with ~1,000 reloads over 90 days using 10 per page so that's good. 

Update 2: I have found an issue where if you change the Timestamp format (as I want it to use 24-hours not AM/PM) then the load fails as there is some code run to determine the reload start before the variables are set which means it uses the previous format in one place and then attempts to use the new format to convert it back to a number and fails. The workaround is to add this (or whatever you want your format to be) to the start of the script once and then you can remove it:

SET TimestampFormat='DD/MM/YYYY hh:mm:ss[.fff]';

[Daniel_Pilla]

Hi @AlexOmetis,

I have addressed both issues above in 1.2.1, now posted on the page. I've created a new variable vu_reloads_page_size to address the message size issue (again unless a single reload is too large), and I've moved the system variables to the top of the "Main" tab to make sure that they are resolved first. Refer to the changelog on this post for details.

Cheers,

[jjdench]

Hi,

Are there any plans to include scheduled Automations details within the Reload Analyzer? Or is that information already available?

Thanks

Justin