QV12 excessive CAL consumption / restart resolved?

[rothtd]

I'm curious if any of you have seen this situation. We are running QV 12.10 SR6 (12.10.20400.0) and a while back we noticed that our usage CALs were being rapidly and abnormally consumed. We normally keep them for overflow purposes, and we were quickly heading toward an outage due to lack of licensing. We have some great analytics on our license consumption and could find nothing abnormal. No sudden spike in Qlik usage globally, and the average session time wasn't increasing. We have a mixture of Named, Session and Usage CALs and re-balancing them (assigning named CALs to individuals consuming most of our session/usage CALs had little to no impact). Just before our usage CALs were completely exhausted we restarted our servers as we could think of nothing else to remedy the situation. After the restart we noticed the excessive consumption had returned to normal, and our Usage CALs started to regenerate back.

During a review of this situation we could find no abnormal or change in user behavior. We did not change our usage patterns (CAL consumption) before, during or after this event. We did notice that during this event we ALWAYS had our Session CALs 100% allocated (hence why we were consuming Usage CALs), even during periods of low system use. Prior to this event the normal pattern was that most of the day you could find a few session CALs left unallocated, and only during peek usage times were they 100% utilized (after our event this pattern returned). Our first guess was that for some reasons our users were having longer sessions - but our log files proved this wrong. Our logs indicated that neither the number of sessions nor the session durations had increased during our "event". This leads me to the theory that QlikView, during the time period of our event, was not releasing Session CALs when the user had ended their session (even though the log files indicated that the users session had ended). One of our system admins stated that he restarted the servers just prior to the event starting which is why he got the idea to restart them a second time (which was the fix). He said when he restarted them the first time he did not stop all QV services first, just restarted the servers. This was a common practice for ~10 years in multiple versions of QV for us (and Qlik support has told us you don't need to stop services prior to shutdown/restart). The second time he restarted he made sure to first stop all QV services, then restart the server.

So - thoughts from all of you. Have you ever experienced this before? Do you normally stop services prior to a restart, or do you just restart the server? Any ideas on what might have been hung up on the QV system side causing the sessions to not be released? Did something change in QV12 where the system is more sensitive to server restarts?

Thanks in advance for your comments!

The below graph depicts what you would expect. A normal pattern of sporadic usage CAL consumption, with our "event" causing excessive and abnormal consumption, and then finally the return to normal as the usage CALs regenerate.

[rothtd · Accepted solution]

Update - after investigation I believe what is happening is that one of our QVS nodes is hanging up periodically. In this situation the QVS Service is not down, but the node is not showing any activity. The assumption is that any user sessions utilizing a Session CAL at the time the QVS node locked up are not released - that is a number of Session CALs are permanently locked by the dead QVS node. This scenario would cause the rapid consumption of usage CALs like I am noticing. I can also confirm that when the QVS node is restarted there are Session CALs that are immediately released back into the pool. I'm going to assume this is the root cause of my rapid Usage CAL consumption issue.

[khalander]

Hi Trevor,

Hope you are doing well.

Even i am facing similar type of issue. Could you please help me to resolve this issue. What are the steps need to follow to resolve this issue.

Thanks in Advance

Regards,

Khalander

[rothtd]

Khalander - This was a difficult one to troubleshoot. I would start with monitoring and trying to understand the situation. In my case, we have a mix of named, session and usage CALs and we were undergoing a situation where one of our QVS clustered nodes would "lock up" requiring a restart. It appeared that when this happened the session cals were not being released back into the pool causing excessive usage cal consumption. I suspect this was happening because prior to the restart all session cals were in use, and when i restarted the problem node some of them immediately freed up. This was nearly a year ago, and I'm again skeptical that this was happening. We have not had this issue reoccur, so I'm not certain a node was actually locking up at all (I have other posts talking about that). I also don't see a bunch of other posts regarding this issue, so it's possible I was not correct in my thinking. While it was true that after a node was restarted several session cals were immediately released back into the pool - this by itself is not a problem, but expected behavior as restarting the server would end active sessions. My assumption that the node was locked up and required a restart may have been in error - I'm still not sure yet. Again, I suspicious because the problem seemed to fix itself (which leads me to believe there wasn't a problem to begin with). It may have also been an issue with one of specific Qlik apps locking up the server - if true this might have explained why it "fixed itself" (if the issue was corrected in the problem app).

I suggest you start with trying to thoroughly understand your issue. Understand your various session timeout settings you have, various log files that help you, and the environment state at the time the problem is occurring (what apps are in use, what users, times of day, load on the servers, etc.), and then contact Qlik Support. Good luck.

- Trevor

[khalander]

Thanks Trevor for your valuable suggestion.

Sure i will check logs.

[peterwh]

Hello,

we had a similar situation when we had a hybrid installation during a migration (Server QV12.20SR6, Plugin on Terminalserver QV12.00SR5). If a user wanted to create a report the plugin failed and the session-CAL was not released after 15min. So all Session-CALs where consumed.

We fixed it through restarts and now our migration is complete and the problem exists no more.

So just a hint, that there could be other problems, who consume more Session-CALs as normal.

Kind regards
Peter