ExtendedException: Type '500' thrown in file 'c:\jws\engine-common-ws\src\server\appmanagement.cpp'

[Huiying]

Hi

ExtendedException: Type '500' thrown in file 'c:\jws\engine-common-ws\src\server\appmanagement.cpp' in function 'CTheServer::GetDataModelMetadataRest_impl' on line '560'. Message: 'Internal Server Error' and additional debug info: 'get metadata'

We have 200-300 above error messages each day from engine services of most nodes. Only 2 newly added nodes don't have that (yet).
Can you give me some hints where to investigate further?

Best regards,
Susan

[rzenere_avvale]

Hey @Huiying,

would you mind sharing more details about your environment? Release, configurations, ...
Personally I haven't found that specific error in the past, but I do remember of a similar issue: Connector Error - connector is not responding

I would suggest to start checking if there is any kind of error on Windows' Event Viewer and in any case check if you did configure properly antivirus exceptions

Riccardo

[Huiying]

Hi Riccardo
Thank you for advise. We use Qlik Sense Enterprise 2020 Feb version in a clustered environment (8 nodes)
Between our QS cluster and external network, we have HA proxy to loadbalance incoming traffic.
Which kind of configuration information is relevant here?
For some reasons, the link to the support article you sent doesn't work. I tried to search from Qlik knowledge base, but found it has different error messages https://support.qlik.com/articles/000042224
From windows event log, I see below error happened right before the error message I posted originally in this thread.

I'll check with our IT colleagues about antivirus exceptions.

Best regards,
Susan

[rzenere_avvale]

Thanks Susan,
I fixed the link (text to display and link were inverted, wops).

Do you have any load balancing rule that loads apps on different nodes? Or anything that performs pre-caching of apps?
It shouldn't be this, but thinking of that 'get metadata'... do you also have the App Metadata Analyzer configured in your environment?

Riccardo

[Huiying]

yes, we do use load balancing rules to load apps to dedicated nodes.
And we also have App Metadata Analyzer..

[rzenere_avvale]

Would you mind checking if the error messages are registered at the same time of the App Metadata Analyzer reload?

Riccardo

[Huiying]

Will check it now.
From another hand, count of the error message keeps increasing, but we reload App Metadata Analyzer only once per day...
In June, count of this error is 4555, July 5283, Aug 6582. This month count is already 2578.

[Huiying]

@rzenere_avvale  Now I'm more convinced that it is related to AppMetadataAnalyzer.
AppMetadataAnalyzer started reload at 7am today. 
During 7:01:02-7:08:10am, I see 240 errors about getting metadata. 
The error messages appear in pair. 
First: 
get metadata: 'class ServerPlugin::ObjectReadError': 'Object read error. Received 404 when getting /qrs/app/611300ea-73e4-4b0e-a610-9bc61403e949/open/full'

Then:
ExtendedException: Type '500' thrown in file 'c:\jws\engine-common-ws\src\server\appmanagement.cpp' in function 'CTheServer::GetDataModelMetadataRest_impl' on line '560'. Message: 'Internal Server Error' and additional debug info: 'get metadata'

AppMetadataAnalyzer tried to retrieve metadata of 40 apps from node2. But the 40 apps are loadbalanced to other nodes. Therefore engine got 'Object read error'. 

From AppMdtadataAnalyzer, I see those apps don't have data.

But  AppMetadataAnalyzer configuration is fairly simple. Difficult to understand why it tried to find those 40 apps from node2.
Do you have any hint? Really appreciate your help!

Best regards,
Susan

[rzenere_avvale]

Hi Susan,

in App Metadata Analyzer, what is the configuration of the vu_central_node_host_name? Bases on your previous messages, I would set the FQDN of the central node, bypassing all external load-balancers
Note that you don't need to load balance any other node for this purpose, central node is able to read all metadata (which is generated on app reload, not at the moment of the request)

Also if you don't have Windows authentication as default for your central node, create a specific virtual proxy, link the virtual proxy to the central proxy and set the prefix in the script (variable vu_virtual_proxy_prefix) 

Let me know if anything changes

Riccardo

[Huiying · Accepted solution]

Now the errors are gone when I reload AppMetadataAnalyzer!! 😄
The app's configuration is correct. I removed rim nodes from virtual proxy's load balancing nodes.
For some apps, I now see data from AppMetadataAnalyzer. But for some apps, I still get 0 RAM footprint, even the app was reloaded today.
Do you have any suggestions?

Best regards,
Susan