1 Reply Latest reply: Nov 9, 2017 10:59 AM by Trevor Roth RSS

    QV12 excessive CAL consumption / restart resolved?

    Trevor Roth

      I'm curious if any of you have seen this situation. We are running QV 12.10 SR6 (12.10.20400.0) and a while back we noticed that our usage CALs were being rapidly and abnormally consumed. We normally keep them for overflow purposes, and we were quickly heading toward an outage due to lack of licensing. We have some great analytics on our license consumption and could find nothing abnormal. No sudden spike in Qlik usage globally, and the average session time wasn't increasing. We have a mixture of Named, Session and Usage CALs and re-balancing them (assigning named CALs to individuals consuming most of our session/usage CALs had little to no impact). Just before our usage CALs were completely exhausted we restarted our servers as we could think of nothing else to remedy the situation. After the restart we noticed the excessive consumption had returned to normal, and our Usage CALs started to regenerate back.

       

      During a review of this situation we could find no abnormal or change in user behavior. We did not change our usage patterns (CAL consumption) before, during or after this event. We did notice that during this event we ALWAYS had our Session CALs 100% allocated (hence why we were consuming Usage CALs), even during periods of low system use. Prior to this event the normal pattern was that most of the day you could find a few session CALs left unallocated, and only during peek usage times were they 100% utilized (after our event this pattern returned). Our first guess was that for some reasons our users were having longer sessions - but our log files proved this wrong. Our logs indicated that neither the number of sessions nor the session durations had increased during our "event". This leads me to the theory that QlikView, during the time period of our event, was not releasing Session CALs when the user had ended their session (even though the log files indicated that the users session had ended). One of our system admins stated that he restarted the servers just prior to the event starting which is why he got the idea to restart them a second time (which was the fix). He said when he restarted them the first time he did not stop all QV services first, just restarted the servers. This was a common practice for ~10 years in multiple versions of QV for us (and Qlik support has told us you don't need to stop services prior to shutdown/restart). The second time he restarted he made sure to first stop all QV services, then restart the server.

       

      So - thoughts from all of you. Have you ever experienced this before? Do you normally stop services prior to a restart, or do you just restart the server? Any ideas on what might have been hung up on the QV system side causing the sessions to not be released? Did something change in QV12 where the system is more sensitive to server restarts?

       

      Thanks in advance for your comments!

       

      The below graph depicts what you would expect. A normal pattern of sporadic usage CAL consumption, with our "event" causing excessive and abnormal consumption, and then finally the return to normal as the usage CALs regenerate.

       

      Usage CALs.png

        • Re: QV12 excessive CAL consumption / restart resolved?
          Trevor Roth

          Update - after investigation I believe what is happening is that one of our QVS nodes is hanging up periodically. In this situation the QVS Service is not down, but the node is not showing any activity. The assumption is that any user sessions utilizing a Session CAL at the time the QVS node locked up are not released - that is a number of Session CALs are permanently locked by the dead QVS node. This scenario would cause the rapid consumption of usage CALs like I am noticing. I can also confirm that when the QVS node is restarted there are Session CALs that are immediately released back into the pool. I'm going to assume this is the root cause of my rapid Usage CAL consumption issue.