Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.
We are excited to share the official release version of the Sense "App Metadata Analyzer" app!
This is a "productized" version of the super app that @Daniel_Pilla and @Levi_Turner posted on Qlik Branch (qs-app-metadata-analyzer).
For a deeper analysis of the application including a demo video, please refer to: https://adminplaybook.qlik-poc.com/docs/asset_management/apps/analyze_app_metadata_analyzer.html
June 2020 Update: In version 2.2.1 and forward of the App Metadata Analyzer, a new Alerting sheet has been added, along with two new variables in the load script. The purpose of this sheet and added capability is to make integration with Qlik Alerting as simple as possible. This new capability and view allows for a Qlik administrator to easily see what applications have breached what thresholds (as well as how many) and be quickly alerted on them. It also allows for the administrator to disable alerts for specific applications and mark others as under review, which might have a different cadence. Please see the attached guide for much more detail and example policies/configurations.
Version History
Sheets:
Dashboard
Threshold Analysis
App Analysis
Alerting
App Availability
Cool, this is really handy thanks so much @Tyler_Waterfall
This looks really useful. I have just downloaded it but it can't find the connector 'monitor_apps_REST_app'. Must be a problem with the Security Rules as the connector is definitely there.
Well done.
We have been using this app for a while. It is useful.
Recently we added 2 customer-facing nodes and loadbalanced all customers to dedicated nodes.
But after that, we found the metadata in new nodes do not show in App Metadata Analyzer.
These are my investigation steps:
- port 443, normal. QS proxy service listening, nothing else is using the port.
- had quick look on logs of the new nodes, didn't (yet) find something special.
- Tried url http(s)://{server}/api/v1/apps/{GUID}/data/metadata. I can see the metadata of apps in new nodes only in new nodes, not in central node. But this seems not stable. Once I indeed see metadata of those apps in central node, then I quickly reloaded the App Metadata Analyzer. But there is still no data of those apps.
Today I plan to check certificates of new nodes, & configuration changes on the 'old' nodes.
I highly appreciate any hint what else should be checked! @Tyler_Waterfall
Best regards,
Susan
Good morning @Huiying ,
personally I had a similar issue in the past, in an environment where I balance apps on specific nodes.
To solve it, I created a new virtual proxy that load balances every node (central + RIMs) and then I linked it to the central proxy. This way (once modified the script to use this virtual proxy) I was able to reload all metadata.
Note that this new virtual proxy is a 'service' one, so no end user is going to use it.
Any other way to solve this kind of issue is very welcome.
I hope this helps,
Riccardo
Thank you Riccardo!
I tested the method, now url (http(s)://{server}/DedicatedVirtualProxy)api/v1/apps/{GUID}/data/metadata) works on central node, but I still got 'Error: HTTP protocol error 500 (Internal Server Error):' from App Metadata Analyzer's reload script log. And the App Metadata Analyzer has count of apps on the server, but it doesn't have RAM data.
Do you have further suggestions?
Thank you for helping!
Best regards,
Susan
Would you mind sharing the complete log? (censored where needed)
Riccardo
Below is one example. What does 'the app has wrong format' mean?
20200617T112938.472+0300 0553 EXIT FOR // We successfully fetched app metadata and can proceed onward. Woohoo!
20200617T112938.475+0300 0569 NEXT i
20200617T112938.475+0300 0366 LET vAppGUID = Peek('AppID',i,'AppReloadedInJune2018Plus')
20200617T112938.476+0300 0367 For a=1 to 3 // QLIK-98225 reload failure without loop logic.
20200617T112938.476+0300 0368 TRACE Working on App 61ffb289-99e5-4611-84f4-6d1dd6a513aa
20200617T112938.476+0300 0368 Working on App 61ffb289-99e5-4611-84f4-6d1dd6a513aa
20200617T112938.476+0300 0369
20200617T112938.476+0300 0369 LIB CONNECT TO 'monitor_apps_REST_app'
20200617T112939.278+0300 Connected.
20200617T112939.280+0300 0371 RestConnectorMasterTable:
20200617T112939.280+0300 0372 SQL SELECT
20200617T112939.280+0300 0373 "static_byte_size",
20200617T112939.280+0300 0374 "__KEY_root",
20200617T112939.280+0300 0375 (SELECT
20200617T112939.280+0300 0376 "cpu_time_spent_ms",
20200617T112939.280+0300 0377 "__KEY_reload_meta",
20200617T112939.280+0300 0378 "__FK_reload_meta",
20200617T112939.280+0300 0379 (SELECT
20200617T112939.280+0300 0380 "logical_cores",
20200617T112939.280+0300 0381 "total_memory",
20200617T112939.280+0300 0382 "__FK_hardware"
20200617T112939.280+0300 0383 FROM "hardware" FK "__FK_hardware")
20200617T112939.280+0300 0384 FROM "reload_meta" PK "__KEY_reload_meta" FK "__FK_reload_meta"),
20200617T112939.280+0300 0385 (SELECT
20200617T112939.280+0300 0386 "name",
20200617T112939.280+0300 0387 "is_system",
20200617T112939.280+0300 0388 "is_hidden",
20200617T112939.280+0300 0389 "is_semantic",
20200617T112939.281+0300 0390 "distinct_only",
20200617T112939.281+0300 0391 "cardinal",
20200617T112939.281+0300 0392 "total_count",
20200617T112939.281+0300 0393 "is_locked",
20200617T112939.281+0300 0394 "always_one_selected",
20200617T112939.281+0300 0395 "is_numeric",
20200617T112939.281+0300 0396 "comment",
20200617T112939.281+0300 0397 "byte_size",
20200617T112939.281+0300 0398 "__KEY_fields",
20200617T112939.281+0300 0399 "__FK_fields",
20200617T112939.281+0300 0400 (SELECT
20200617T112939.281+0300 0401 "@Value",
20200617T112939.281+0300 0402 "__FK_src_tables"
20200617T112939.281+0300 0403 FROM "src_tables" FK "__FK_src_tables" ArrayValueAlias "@Value"),
20200617T112939.281+0300 0404 (SELECT
20200617T112939.281+0300 0405 "@Value" AS "@Value_u0",
20200617T112939.281+0300 0406 "__FK_tags"
20200617T112939.281+0300 0407 FROM "tags" FK "__FK_tags" ArrayValueAlias "@Value_u0")
20200617T112939.281+0300 0408 FROM "fields" PK "__KEY_fields" FK "__FK_fields"),
20200617T112939.281+0300 0409 (SELECT
20200617T112939.281+0300 0410 "name" AS "name_u0",
20200617T112939.281+0300 0411 "is_system" AS "is_system_u0",
20200617T112939.281+0300 0412 "is_semantic" AS "is_semantic_u0",
20200617T112939.281+0300 0413 "is_loose",
20200617T112939.281+0300 0414 "no_of_rows",
20200617T112939.281+0300 0415 "no_of_fields",
20200617T112939.281+0300 0416 "no_of_key_fields",
20200617T112939.281+0300 0417 "comment" AS "comment_u0",
20200617T112939.281+0300 0418 "byte_size" AS "byte_size_u0",
20200617T112939.281+0300 0419 "__FK_tables"
20200617T112939.281+0300 0420 FROM "tables" FK "__FK_tables")
20200617T112939.281+0300 0421 FROM JSON (wrap on) "root" PK "__KEY_root"
20200617T112939.281+0300 0422 WITH CONNECTION (
20200617T112939.281+0300 0423 URL "https://server/ama/api/v1/apps/61ffb289-99e5-4611-84f4-6d1dd6a513aa/data/metadata",
20200617T112939.281+0300 0424 HTTPHEADER "User-Agent" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36"
20200617T112939.281+0300 0425 )
20200617T112939.732+0300 37 fields found: logical_cores, total_memory, __FK_hardware, cpu_time_spent_ms, __KEY_reload_meta, __FK_reload_meta, @Value, __FK_src_tables, @Value_u0, __FK_tags, name, is_system, is_hidden, is_semantic, distinct_only, cardinal, total_count, is_locked, always_one_selected, is_numeric, comment, byte_size, __KEY_fields, __FK_fields, name_u0, is_system_u0, is_semantic_u0, is_loose, no_of_rows, no_of_fields, no_of_key_fields, comment_u0, byte_size_u0, __FK_tables, static_byte_size, __KEY_root, __extra_,
20200617T112939.736+0300 Error: HTTP protocol error 500 (Internal Server Error):
20200617T112939.736+0300
20200617T112939.736+0300
20200617T112939.737+0300 0427 IF ScriptError<>12 THEN // Successfully found app metadata. If ScriptError = 12, there was an issue fetching metadata
20200617T112939.748+0300 0556 DROP TABLE RestConnectorMasterTable
20200617T112939.748+0300 0557 TRACE Script Error = 12: File 61ffb289-99e5-4611-84f4-6d1dd6a513aa has wrong format
20200617T112939.748+0300 0557 Script Error = 12: File 61ffb289-99e5-4611-84f4-6d1dd6a513aa has wrong format
20200617T112939.748+0300 0558
20200617T112939.749+0300 0558 IF a<3 THEN // QLIK-98225 reload failure without loop logic.
20200617T112939.749+0300 0559 TRACE 'Trying again...'
20200617T112939.749+0300 0559 'Trying again...'
20200617T112939.749+0300 0560
20200617T112939.749+0300 0560 ELSEIF a=3 THEN
20200617T112939.750+0300 0567 END IF
20200617T112939.750+0300 0568 Next a // loop for error handling
20200617T112939.750+0300 0368 TRACE Working on App 61ffb289-99e5-4611-84f4-6d1dd6a513aa
20200617T112939.750+0300 0368 Working on App 61ffb289-99e5-4611-84f4-6d1dd6a513aa
20200617T112939.750+0300 0369
20200617T112939.750+0300 0369 LIB CONNECT TO 'monitor_apps_REST_app'
20200617T112940.410+0300 Connected.
20200617T112940.413+0300 0371 RestConnectorMasterTable:
20200617T112940.413+0300 0372 SQL SELECT
20200617T112940.413+0300 0373 "static_byte_size",
20200617T112940.413+0300 0374 "__KEY_root",
20200617T112940.413+0300 0375 (SELECT
20200617T112940.413+0300 0376 "cpu_time_spent_ms",
20200617T112940.413+0300 0377 "__KEY_reload_meta",
20200617T112940.413+0300 0378 "__FK_reload_meta",
20200617T112940.413+0300 0379 (SELECT
20200617T112940.413+0300 0380 "logical_cores",
20200617T112940.413+0300 0381 "total_memory",
20200617T112940.413+0300 0382 "__FK_hardware"
20200617T112940.413+0300 0383 FROM "hardware" FK "__FK_hardware")
20200617T112940.413+0300 0384 FROM "reload_meta" PK "__KEY_reload_meta" FK "__FK_reload_meta"),
20200617T112940.413+0300 0385 (SELECT
20200617T112940.413+0300 0386 "name",
20200617T112940.413+0300 0387 "is_system",
20200617T112940.413+0300 0388 "is_hidden",
20200617T112940.413+0300 0389 "is_semantic",
20200617T112940.413+0300 0390 "distinct_only",
20200617T112940.413+0300 0391 "cardinal",
20200617T112940.413+0300 0392 "total_count",
20200617T112940.413+0300 0393 "is_locked",
20200617T112940.413+0300 0394 "always_one_selected",
20200617T112940.413+0300 0395 "is_numeric",
20200617T112940.413+0300 0396 "comment",
20200617T112940.413+0300 0397 "byte_size",
20200617T112940.413+0300 0398 "__KEY_fields",
20200617T112940.413+0300 0399 "__FK_fields",
20200617T112940.413+0300 0400 (SELECT
20200617T112940.413+0300 0401 "@Value",
20200617T112940.413+0300 0402 "__FK_src_tables"
20200617T112940.413+0300 0403 FROM "src_tables" FK "__FK_src_tables" ArrayValueAlias "@Value"),
20200617T112940.413+0300 0404 (SELECT
20200617T112940.413+0300 0405 "@Value" AS "@Value_u0",
20200617T112940.413+0300 0406 "__FK_tags"
20200617T112940.413+0300 0407 FROM "tags" FK "__FK_tags" ArrayValueAlias "@Value_u0")
20200617T112940.413+0300 0408 FROM "fields" PK "__KEY_fields" FK "__FK_fields"),
20200617T112940.413+0300 0409 (SELECT
20200617T112940.413+0300 0410 "name" AS "name_u0",
20200617T112940.413+0300 0411 "is_system" AS "is_system_u0",
20200617T112940.413+0300 0412 "is_semantic" AS "is_semantic_u0",
20200617T112940.413+0300 0413 "is_loose",
20200617T112940.413+0300 0414 "no_of_rows",
20200617T112940.413+0300 0415 "no_of_fields",
20200617T112940.413+0300 0416 "no_of_key_fields",
20200617T112940.413+0300 0417 "comment" AS "comment_u0",
20200617T112940.413+0300 0418 "byte_size" AS "byte_size_u0",
20200617T112940.413+0300 0419 "__FK_tables"
20200617T112940.413+0300 0420 FROM "tables" FK "__FK_tables")
20200617T112940.413+0300 0421 FROM JSON (wrap on) "root" PK "__KEY_root"
20200617T112940.413+0300 0422 WITH CONNECTION (
20200617T112940.413+0300 0423 URL "https://server/ama/api/v1/apps/61ffb289-99e5-4611-84f4-6d1dd6a513aa/data/metadata",
20200617T112940.413+0300 0424 HTTPHEADER "User-Agent" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36"
20200617T112940.413+0300 0425 )
20200617T112940.883+0300 37 fields found: logical_cores, total_memory, __FK_hardware, cpu_time_spent_ms, __KEY_reload_meta, __FK_reload_meta, @Value, __FK_src_tables, @Value_u0, __FK_tags, name, is_system, is_hidden, is_semantic, distinct_only, cardinal, total_count, is_locked, always_one_selected, is_numeric, comment, byte_size, __KEY_fields, __FK_fields, name_u0, is_system_u0, is_semantic_u0, is_loose, no_of_rows, no_of_fields, no_of_key_fields, comment_u0, byte_size_u0, __FK_tables, static_byte_size, __KEY_root, __extra_,
20200617T112940.893+0300 Error: HTTP protocol error 500 (Internal Server Error):
20200617T112940.893+0300
20200617T112940.893+0300
20200617T112940.895+0300 0427 IF ScriptError<>12 THEN // Successfully found app metadata. If ScriptError = 12, there was an issue fetching metadata
20200617T112940.905+0300 0556 DROP TABLE RestConnectorMasterTable
20200617T112940.905+0300 0557 TRACE Script Error = 12: File 61ffb289-99e5-4611-84f4-6d1dd6a513aa has wrong format
20200617T112940.905+0300 0557 Script Error = 12: File 61ffb289-99e5-4611-84f4-6d1dd6a513aa has wrong format
20200617T112940.905+0300 0558
20200617T112940.905+0300 0558 IF a<3 THEN // QLIK-98225 reload failure without loop logic.
20200617T112940.906+0300 0559 TRACE 'Trying again...'
20200617T112940.906+0300 0559 'Trying again...'
20200617T112940.906+0300 0560
20200617T112940.906+0300 0560 ELSEIF a=3 THEN
20200617T112940.906+0300 0567 END IF
20200617T112940.907+0300 0568 Next a // loop for error handling
20200617T112940.907+0300 0368 TRACE Working on App 61ffb289-99e5-4611-84f4-6d1dd6a513aa
20200617T112940.907+0300 0368 Working on App 61ffb289-99e5-4611-84f4-6d1dd6a513aa
20200617T112940.907+0300 0369
20200617T112940.907+0300 0369 LIB CONNECT TO 'monitor_apps_REST_app'
20200617T112941.538+0300 Connected.
20200617T112941.541+0300 0371 RestConnectorMasterTable:
20200617T112941.541+0300 0372 SQL SELECT
20200617T112941.541+0300 0373 "static_byte_size",
20200617T112941.541+0300 0374 "__KEY_root",
20200617T112941.541+0300 0375 (SELECT
20200617T112941.541+0300 0376 "cpu_time_spent_ms",
20200617T112941.541+0300 0377 "__KEY_reload_meta",
20200617T112941.541+0300 0378 "__FK_reload_meta",
20200617T112941.541+0300 0379 (SELECT
20200617T112941.541+0300 0380 "logical_cores",
20200617T112941.541+0300 0381 "total_memory",
20200617T112941.541+0300 0382 "__FK_hardware"
20200617T112941.541+0300 0383 FROM "hardware" FK "__FK_hardware")
20200617T112941.541+0300 0384 FROM "reload_meta" PK "__KEY_reload_meta" FK "__FK_reload_meta"),
20200617T112941.541+0300 0385 (SELECT
20200617T112941.541+0300 0386 "name",
20200617T112941.541+0300 0387 "is_system",
20200617T112941.541+0300 0388 "is_hidden",
20200617T112941.541+0300 0389 "is_semantic",
20200617T112941.541+0300 0390 "distinct_only",
20200617T112941.541+0300 0391 "cardinal",
20200617T112941.541+0300 0392 "total_count",
20200617T112941.541+0300 0393 "is_locked",
20200617T112941.541+0300 0394 "always_one_selected",
20200617T112941.541+0300 0395 "is_numeric",
20200617T112941.541+0300 0396 "comment",
20200617T112941.541+0300 0397 "byte_size",
20200617T112941.541+0300 0398 "__KEY_fields",
20200617T112941.541+0300 0399 "__FK_fields",
20200617T112941.541+0300 0400 (SELECT
20200617T112941.541+0300 0401 "@Value",
20200617T112941.541+0300 0402 "__FK_src_tables"
20200617T112941.541+0300 0403 FROM "src_tables" FK "__FK_src_tables" ArrayValueAlias "@Value"),
20200617T112941.541+0300 0404 (SELECT
20200617T112941.541+0300 0405 "@Value" AS "@Value_u0",
20200617T112941.541+0300 0406 "__FK_tags"
20200617T112941.541+0300 0407 FROM "tags" FK "__FK_tags" ArrayValueAlias "@Value_u0")
20200617T112941.541+0300 0408 FROM "fields" PK "__KEY_fields" FK "__FK_fields"),
20200617T112941.541+0300 0409 (SELECT
20200617T112941.541+0300 0410 "name" AS "name_u0",
20200617T112941.541+0300 0411 "is_system" AS "is_system_u0",
20200617T112941.541+0300 0412 "is_semantic" AS "is_semantic_u0",
20200617T112941.541+0300 0413 "is_loose",
20200617T112941.541+0300 0414 "no_of_rows",
20200617T112941.541+0300 0415 "no_of_fields",
20200617T112941.541+0300 0416 "no_of_key_fields",
20200617T112941.541+0300 0417 "comment" AS "comment_u0",
20200617T112941.541+0300 0418 "byte_size" AS "byte_size_u0",
20200617T112941.541+0300 0419 "__FK_tables"
20200617T112941.541+0300 0420 FROM "tables" FK "__FK_tables")
20200617T112941.541+0300 0421 FROM JSON (wrap on) "root" PK "__KEY_root"
20200617T112941.541+0300 0422 WITH CONNECTION (
20200617T112941.541+0300 0423 URL "https://server/ama/api/v1/apps/61ffb289-99e5-4611-84f4-6d1dd6a513aa/data/metadata",
20200617T112941.541+0300 0424 HTTPHEADER "User-Agent" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36"
20200617T112941.541+0300 0425 )
20200617T112941.988+0300 37 fields found: logical_cores, total_memory, __FK_hardware, cpu_time_spent_ms, __KEY_reload_meta, __FK_reload_meta, @Value, __FK_src_tables, @Value_u0, __FK_tags, name, is_system, is_hidden, is_semantic, distinct_only, cardinal, total_count, is_locked, always_one_selected, is_numeric, comment, byte_size, __KEY_fields, __FK_fields, name_u0, is_system_u0, is_semantic_u0, is_loose, no_of_rows, no_of_fields, no_of_key_fields, comment_u0, byte_size_u0, __FK_tables, static_byte_size, __KEY_root, __extra_,
20200617T112941.994+0300 Error: HTTP protocol error 500 (Internal Server Error):
20200617T112941.994+0300
20200617T112941.994+0300
20200617T112941.995+0300 0427 IF ScriptError<>12 THEN // Successfully found app metadata. If ScriptError = 12, there was an issue fetching metadata
20200617T112942.009+0300 0556 DROP TABLE RestConnectorMasterTable
20200617T112942.010+0300 0557 TRACE Script Error = 12: File 61ffb289-99e5-4611-84f4-6d1dd6a513aa has wrong format
20200617T112942.010+0300 0557 Script Error = 12: File 61ffb289-99e5-4611-84f4-6d1dd6a513aa has wrong format
20200617T112942.010+0300 0558
20200617T112942.010+0300 0558 IF a<3 THEN // QLIK-98225 reload failure without loop logic.
20200617T112942.011+0300 0561 AppEndpointsNotFound:
20200617T112942.011+0300 0562 LOAD
20200617T112942.011+0300 0563 '61ffb289-99e5-4611-84f4-6d1dd6a513aa' AS AppID,
20200617T112942.011+0300 0564 1 AS AppEndpointNotFound
20200617T112942.011+0300 0565 AUTOGENERATE(1)
20200617T112942.011+0300 2 fields found: AppID, AppEndpointNotFound,
20200617T112942.012+0300 1 lines fetched
20200617T112942.012+0300 0566 END IF
20200617T112942.012+0300 0567 END IF
20200617T112942.013+0300 0568 Next a // loop for error handling
@Levi_Turner
Could you give a hint where to check for the 'HTTP protocol error 500' message in App Metadata Analyzer reload script log?
We use Qlik Sense Enterprise 2020 Feb version on a 8-node cluster. We check App Metadata Analyzer every now and then.
2 weeks ago, we added 2 customer-facing node, and loadbalanced all customers to own dedicated customer-facing node.
This week we found that for the 2 new nodes & their reload node, App Metadata Analyzer has app count but doesn't have RAM data.
URL (http(s)://{central node}/{dedicated virtual proxy}/api/v1/apps/{GUID}/data/metadata) works fine from browser.
The whole cluster works normally. New nodes also work normally from both hub & mashup.
My investigation steps:
- port 443: normal, no blocker, listening by QS proxy service
- certificates: normal
- created dedicated virtual proxy for App Metadata Analyzer. The virtual proxy loadbalances all RIM nodes. After reload, error is still in the script log.
- log: quickly went through logs on the new nodes, but didn't find yet something I should focus on now.
I feel out of ideas where to check next. Highly appreciate any help!
Best regards,
Susan
Good morning @Huiying ,
just to be sure:
- created dedicated virtual proxy for App Metadata Analyzer. The virtual proxy loadbalances all RIM nodes. After reload, error is still in the script log.
Is the new virtual proxy load balancing all RIM nodes AND central node?
If not, for what I've seen in the past, it's possible to have different results from browser and script call (also the response can change from time to time)
Riccardo
@rzenere_avvale yes, the dedicated virtual proxy load balances all 8 nodes in the cluster, including central node, proxy node, customer-facing nodes & reload nodes.
When checking the url in browser, I agree that it is not stable. I sometimes get it connected, sometimes not.
But for script, it has never worked I think, even when url works