CQvXmlInterfaceRequestHandler errors, Inconsistencies, PGO Problems, & QVS Crashes

[#1]

Hello everybody,

I am new to managing QlikView and I am working on an environment that has some sporadic issues. I feel like I am hitting a wall in terms of new ideas to try, so I'm posting this in hopes that somebody might have a few suggestions.

First, a bit of info on our architecture. We run a clustered environment with the following roles, all running QV 11.20 SR 6 on Windows Server 2012:

Server 1: Publisher & QMC

Server 2: QVS, DSC: & Access Point

Server 3: QVS

File Server Cluster (2 servers): Presents SAN CIFS share, which is our document root. Path to root is defined as \\<ipaddress>\qvprd.

SAN: Hosts qvprd share as well as other non-QlikView shares.

Our systems appear to be over-sized for current performance needs, so I wouldn't think this is a resource issue. All have 512 GB RAM and 4x 6-core processors at 2.89GHz (24 cores total). I watch these systems closely and the QVS servers never break a sweat and almost always are below 60% memory allocation. The Publisher server is even lower, with heavy CPU utilization only during our busy reload times (3am-6am). We currently have a maximum of 12 reload engines running concurrently.

All systems run System Center Endpoint Protection and have exclusions for QlikView-related folders and files. And I have confirmed that backups are not running during the times that we experience problems.

Two or three times a week, we will encounter an error that affects QVS on Servers 2 & 3. Most of the time the errors we see start off on one of the two servers with the following:

1. CQvXmlInterfaceRequestHandler - Catch: Threw an error...

This is accompanied by the "No Server" message in Access Point, and our only recourse is to restart QVS on that system, which resolves the issue immediately. Then, within a few minutes, we usually see either the same error message on the other QVS server, or we see an inconsistency (usually type D, but sometimes type F):

1. Restart: Server aborted trying to recover by restart. Reason for restart: Internal inconsistency, type F, detected.
2. qvpx: Exception while handling request

This usually seems to happen between 8:00 and 10:00 AM. Most of our reloads finish up by 6:00 AM, so there's not a lot of action on these systems outside of the QVS services. At any given time, we usually have between 20 and 35 concurrent users spread across the two systems.

To make things more confusing, we get PGO errors every week or two. We'll have extension-less 0KB files named things like 'CalID' appear in the root folder and C:\ProgramData\QlikTech\QlikViewServer\. The error message in the logs usually have invalid characters in the path, like the message below. These are always accompanied by a QVS crash or restart:

1. PGO: Failed to open C:\P\CalD鮵ꁑ. Error : C:\P\CalD鮵ꁑ contains an incorrect path.. Time: 0 ms

So far, the way we have dealt with those problems is to take a QlikView outage, clean up the PGO files, and restart services.That usually sets us straight for another week or two (max).

I will attach some logs from both server 1 and 2 from this morning, in which we had the exact issue described above. I'm hoping that somebody may have some insight into this problem or suggestions on what to look at. I'm running out of ideas, and I'm really struggling to correlate this to any one thing. There are some weeks where it'll happen 3 times, and some weeks where it happens maybe once. And other than these random crashes, our system runs like a champ. It just has this annoying hiccup.

Anybody have any ideas? I'm all ears.

[Giuseppe_Novello]

Josh,

I notice you have done a sustensial work troubleshooting it. I would suggest to open a support ticket, you might ( be clear you "might" not "are") hitting a PGO issue/defect. In that case Support needs little troubleshoot it, but I would also look into upgrading to the latest version of Qlikview which is V11.20.12742 or V11.20 SR10.

If you submit the support ticket, make sure to document exactly what you have done so far ( do not put "please see this community link", that will slow little the process)  Provide all the Qlikview logs ( server, pub, QMC, etc) and Windows event logs and if you have perform a memory CPU analysis, please attach that as well.

Support Contact:

http://www.qlik.com/us/services/support/product-support

Also, for any chance did you had Qlikview 9 or 10 previously on those servers?

BR

Gio

[#3]

Thanks for your response, Giuseppe. In answer to your question - no, QlikView 9 or 10 were not installed on these systems. The entire environment was rebuilt and redesigned from a single-tier node into this structure we have today, prior to when I came on board at this company. So it was a completely fresh build on brand new hardware.

Opening a support case is definitely on my radar, as is SR 10. I had assumed (maybe incorrectly, I won't know until I try) that the first step that support will recommend is to upgrade to SR10. Since we  have a formalized upgrade process in which our development teams will require testing of all reports in dev prior to upgrading to production, I was hoping to exhaust all potential troubleshooting avenues prior to requesting resources from other teams. Thus my question to the community. I thought maybe I'd encounter others who dealt with a similar issue.

I have looked through the SR 10 release notes and while I see a few bug fixes that look slightly promising/interesting (64333-SR7, 67852-SR9) but nothing that lines up with exactly what we're seeing. So who knows.

Also, I should have mentioned that we do not see the same issue in our development tier, which is a single-server node (QVS, QMC, DSC, Access Point). This led us to believe that the problem might be with QV clustering. But at this point I don't have any evidence to back that theory up.

[Giuseppe_Novello]

Josh,

Thanks! Well SR6 it is currectly outside the patching scope from R&D. Now, seeing that clearing the PGO fix the issue most likely could be a issue that is going on with PGOs ( again is not a confirmation this is the case, it needs a full investigation to confirm) which SR10 have a patch that fix them. However, you must be in SR10 to get the patch.

BR

Gio

[Bill_Britt · Accepted solution]

Hi,

Have to agree with Giuseppe, and you need to upgrade to SR10 and you will need the patches 72745 and 72746.

Bill

[#6]

Sounds good - thank you both for your input!

[#7]

Hi Josh,

We are encountering the same issue and we are on 11.20 SR10. The environment is also clustered environment.

Wondering if you managed to get around this issue please?

Thanks

elly

[Bill_Britt]

Have you asked for and applied patch 72746?

Bill

[#9]

Thanks Bill. Sorry I missed your earlier response to Josh re the patches.

I will ask for the patches and try that out.

Thanks

elly

[Bill_Britt]

I would also get the 72745 patch also. You would install 72745 and then 72746

Bill