topic Re: ExtendedException: Type '500' thrown in file 'c:\jws\engine-common-ws\src\server\appmanagement.c in Management & Governance

ExtendedException: Type '500' thrown in file 'c:\jws\engine-common-ws\src\server\appmanagement.cpp'

Huiying — Thu, 10 Sep 2020 09:07:56 GMT

ExtendedException: Type '500' thrown in file 'c:\jws\engine-common-ws\src\server\appmanagement.cpp' in function 'CTheServer::GetDataModelMetadataRest_impl' on line '560'. Message: 'Internal Server Error' and additional debug info: 'get metadata'

We have 200-300 above error messages each day from engine services of most nodes. Only 2 newly added nodes don't have that (yet).
Can you give me some hints where to investigate further?

Best regards,
Susan

Re: ExtendedException: Type '500' thrown in file 'c:\jws\engine-common-ws\src\server\appmanagement.c

rzenere_avvale — Thu, 10 Sep 2020 11:52:04 GMT

Hey @Huiying,

would you mind sharing more details about your environment? Release, configurations, ...
Personally I haven't found that specific error in the past, but I do remember of a similar issue: Connector Error - connector is not responding

I would suggest to start checking if there is any kind of error on Windows' Event Viewer and in any case check if you did configure properly antivirus exceptions

Riccardo

Re: ExtendedException: Type '500' thrown in file 'c:\jws\engine-common-ws\src\server\appmanagement.c

Huiying — Thu, 10 Sep 2020 11:44:37 GMT

Hi Riccardo
Thank you for advise. We use Qlik Sense Enterprise 2020 Feb version in a clustered environment (8 nodes)
Between our QS cluster and external network, we have HA proxy to loadbalance incoming traffic.
Which kind of configuration information is relevant here?
For some reasons, the link to the support article you sent doesn't work. I tried to search from Qlik knowledge base, but found it has different error messages https://support.qlik.com/articles/000042224
From windows event log, I see below error happened right before the error message I posted originally in this thread.

I'll check with our IT colleagues about antivirus exceptions.

Best regards,
Susan

Re: ExtendedException: Type '500' thrown in file 'c:\jws\engine-common-ws\src\server\appmanagement.c

rzenere_avvale — Thu, 10 Sep 2020 11:55:25 GMT

Thanks Susan,
I fixed the link (text to display and link were inverted, wops).

Do you have any load balancing rule that loads apps on different nodes? Or anything that performs pre-caching of apps?
It shouldn't be this, but thinking of that 'get metadata'... do you also have the App Metadata Analyzer configured in your environment?

Riccardo

Re: ExtendedException: Type '500' thrown in file 'c:\jws\engine-common-ws\src\server\appmanagement.c

Huiying — Thu, 10 Sep 2020 12:06:24 GMT

yes, we do use load balancing rules to load apps to dedicated nodes.
And we also have App Metadata Analyzer..

Re: ExtendedException: Type '500' thrown in file 'c:\jws\engine-common-ws\src\server\appmanagement.c

rzenere_avvale — Thu, 10 Sep 2020 12:08:53 GMT

Would you mind checking if the error messages are registered at the same time of the App Metadata Analyzer reload?

Riccardo

Re: ExtendedException: Type '500' thrown in file 'c:\jws\engine-common-ws\src\server\appmanagement.c

Huiying — Thu, 10 Sep 2020 12:23:08 GMT

Will check it now.
From another hand, count of the error message keeps increasing, but we reload App Metadata Analyzer only once per day...
In June, count of this error is 4555, July 5283, Aug 6582. This month count is already 2578.

Re: ExtendedException: Type '500' thrown in file 'c:\jws\engine-common-ws\src\server\appmanagement.c

Huiying — Fri, 11 Sep 2020 09:48:38 GMT

@rzenere_avvale Now I'm more convinced that it is related to AppMetadataAnalyzer.
AppMetadataAnalyzer started reload at 7am today.
During 7:01:02-7:08:10am, I see 240 errors about getting metadata.
The error messages appear in pair.
First:
get metadata: 'class ServerPlugin::ObjectReadError': 'Object read error. Received 404 when getting /qrs/app/611300ea-73e4-4b0e-a610-9bc61403e949/open/full'

Then:
ExtendedException: Type '500' thrown in file 'c:\jws\engine-common-ws\src\server\appmanagement.cpp' in function 'CTheServer::GetDataModelMetadataRest_impl' on line '560'. Message: 'Internal Server Error' and additional debug info: 'get metadata'

AppMetadataAnalyzer tried to retrieve metadata of 40 apps from node2. But the 40 apps are loadbalanced to other nodes. Therefore engine got 'Object read error'.

From AppMdtadataAnalyzer, I see those apps don't have data.

But AppMetadataAnalyzer configuration is fairly simple. Difficult to understand why it tried to find those 40 apps from node2.
Do you have any hint? Really appreciate your help!

Best regards,
Susan

Re: ExtendedException: Type '500' thrown in file 'c:\jws\engine-common-ws\src\server\appmanagement.c

rzenere_avvale — Fri, 11 Sep 2020 10:05:14 GMT

Hi Susan,

in App Metadata Analyzer, what is the configuration of the vu_central_node_host_name? Bases on your previous messages, I would set the FQDN of the central node, bypassing all external load-balancers
Note that you don't need to load balance any other node for this purpose, central node is able to read all metadata (which is generated on app reload, not at the moment of the request)

Also if you don't have Windows authentication as default for your central node, create a specific virtual proxy, link the virtual proxy to the central proxy and set the prefix in the script (variable vu_virtual_proxy_prefix)

Let me know if anything changes

Riccardo

Re: ExtendedException: Type '500' thrown in file 'c:\jws\engine-common-ws\src\server\appmanagement.c

Huiying — Fri, 11 Sep 2020 12:42:10 GMT

Now the errors are gone when I reload AppMetadataAnalyzer!! 😄
The app's configuration is correct. I removed rim nodes from virtual proxy's load balancing nodes.
For some apps, I now see data from AppMetadataAnalyzer. But for some apps, I still get 0 RAM footprint, even the app was reloaded today.
Do you have any suggestions?

Best regards,
Susan

Re: ExtendedException: Type '500' thrown in file 'c:\jws\engine-common-ws\src\server\appmanagement.c

Huiying — Mon, 14 Sep 2020 08:21:13 GMT

Now AppMetadataAnalyzer has all data. Issue resolved.
Here is summary of this issue.
The error messages appear because AppMetadataAnalyzer tried to retrieve app metadata from rim node where apps were not loadbalanced to.
Solution: remove the rim nodes from AppMetadataAnalyzer's virtual proxy's load balancing nodes list.

Big thank to @rzenere_avvale !!

Re: ExtendedException: Type '500' thrown in file 'c:\jws\engine-common-ws\src\server\appmanagement.c

rzenere_avvale — Mon, 14 Sep 2020 08:51:33 GMT

Hey Susan,

great that the errors are not showing anymore. I'm still confused why you had an incremental number of errors, but having solved the issue is still good enough.
Unluckily no, I don't have anymore clues about the 0RAM footprint. I would suggest to try to perform manually the same extraction as Qlik does (you should be able to see in the log file the precise URL that Qlik uses)

Riccardo

Re: ExtendedException: Type '500' thrown in file 'c:\jws\engine-common-ws\src\server\appmanagement.c

Huiying — Mon, 14 Sep 2020 11:33:31 GMT

I assume increase in number of error probably because we constantly have new customers & new apps..
What do you mean by 'perform manually the same extraction'?

Re: ExtendedException: Type '500' thrown in file 'c:\jws\engine-common-ws\src\server\appmanagement.c

rzenere_avvale — Mon, 14 Sep 2020 12:14:12 GMT

If you export the reload log, you should find a part of the script where a call like this is performed:

Note: in your log you (probably) won't find anything with win, because it is the prefix of my Windows virtual proxy.

Find the row with one of those apps that have no footprint/... and, from your browser, call the same url.
This way you should be able to obtain the same metadata as Qlik does while reloading.

At this point, I can think to three scenarios:

The call returns no metadata (I would expect an error): no reload has been performed for the app (or, but I'm not sure about this one, it has been imported from another environment)
The call returns metadata but no info about tables/fields is there: that could be an app that has been reloaded in your environment but at the end of the reload had no data (i.e. an ETL which reads from a source and then stores data in QVD format and drops it from the app)
The call returns metadata and you have all the info: this should be fine for App Metadata Analyzer

Riccardo

Re: ExtendedException: Type '500' thrown in file 'c:\jws\engine-common-ws\src\server\appmanagement.c

Huiying — Fri, 18 Sep 2020 07:05:11 GMT

Thank you Riccardo
I'll use this method next time when AppMetadataAnalyzer gives 0 RAM footprint.
Big thank to your help! 🙂

Best regards,
Susan