QlikView Performance Monitoring

    Built upon the application I developed and published under the title Processing Windows Perfmon Logs using QlikView  I've expanded on that to specifically monitor and analyse the performance of QlikView Servers (now that I manage several QlikView Environments). It is a one stop shop for viewing the Performance of the QlikView servers (using Windows Perfmon) and also analysing the various QlikView Logs. I know the QlikView System Monitor will provide some of the same functionality but I feel my version is far more comprehensive. I recommend setting up Windows Performance Monitoring at the earliest opportunity, don't wait until you experience issues as the chance to collect valuable data will have been lost.

     

    As a prerequisite I try to ensure that all the Windows Perfmon Logs I collect are of a uniform nature:

     

             Have a naming convention: servername_YYYYMMDD

             Logs are saved as 'csv' files

             Collect metrics at 5 minute intervals*

             Run from midnight (00:00.01) to midnight (23:55.01)

     

    *After some consideration I decided that 5 minute intervals were about the right frequency.

     

    Attachments:

    1) The 'QV_Utilisation_Reporting.qvw' app is the front end application for displaying the processed QVD's. I've set it to load the past 3 months data only. There's a large number of dashboards and charts that will suddenly take shape when the processed data is loaded in to it, though some may need tweeking to best display your data.

     

    2) The 'QV_Perfmon_processing.qvw' app processes the Windows Perfmon (csv) Logs.

     

    3) The 'QV_Log_processing.qvw' app processes the Audit, Events, Performance, Session and Weblogs. The Audit Logs are not loaded in to the front-end ('QV_Utilisation_Reporting.qvw') app but can be viewed and manipulated using this app giving greater functionality and granuality than the raw logs.

     

    4) The 'QV-IIS_Logs_processing.qvw' app processes the IIS Logs. To be used if IIS is used instead of QlikView Web Server.

     

    5) The 'QV_Reload-Log_processing.qvw' processes the Reload Logs on the Publisher server and calculates the Reload & Distribution time. This may be specific to our organisation only.

     

    6) The 'QV_TODAY_processing.qvw' processes all of the above logs for today only and saves them to files that will be over-written on each reload. This is to provide at up to the minute view of Performance by reloading on a scheduled or ad-hoc basic just before the 'QV_Utilisation_Reporting.qvw' is reloaded.

     

    7) The 'Perfmon-QlikView.xml' file is a template which can be used to set up Perfmon collectors. The Appendix also lists the relevant metrics.

     

    Appendix:

    For QlikView Servers I recommend the following Metrics be collected

     

    \.NET CLR Memory(QVDistributionService)\# of Pinned Objects

    \LogicalDisk(D:)\Avg. Disk Bytes/Transfer

    \LogicalDisk(D:)\Avg. Disk sec/Read

    \LogicalDisk(D:)\Avg. Disk sec/Write

    \LogicalDisk(D:)\Current Disk Queue Length

    \LogicalDisk(D:)\% Free Space

    \LogicalDisk(D:)\Free Megabytes

    \Memory\% Committed Bytes In Use

    \Memory\Available MBytes

    \Memory\Page Faults/sec

    \Memory\Page Reads/sec

    \Memory\Pages/sec

    \Memory\Pool Nonpaged Bytes

    \Network Interface(*)\Bytes Total/sec

    \Network Interface(*)\Current Bandwidth

    \Paging File(_Total)\% Usage

    \PhysicalDisk(_Total)\% Disk Time

    \PhysicalDisk(_Total)\Avg. Disk Bytes/Transfer

    \PhysicalDisk(_Total)\Avg. Disk Queue Length

    \Process(QVDistributionService)\% Processor Time

    \Process(QVDistributionService)\Private Bytes

    \Process(QVDistributionService)\Thread Count

    \Process(QVDistributionService)\Virtual Bytes

    \Process(QVDistributionService)\Working Set

    \Process(QVDirectoryServiceConnector)\% Processor Time

    \Process(QVDirectoryServiceConnector)\Private Bytes

    \Process(QVDirectoryServiceConnector)\Thread Count

    \Process(QVDirectoryServiceConnector)\Virtual Bytes

    \Process(QVDirectoryServiceConnector)\Working Set

    \Process(QVManagementService)\% Processor Time

    \Process(QVManagementService)\Private Bytes

    \Process(QVManagementService)\Thread Count

    \Process(QVManagementService)\Virtual Bytes

    \Process(QVManagementService)\Working Set

    \Process(qvs)\% Processor Time

    \Process(qvs)\Private Bytes

    \Process(qvs)\Thread Count

    \Process(qvs)\Virtual Bytes

    \Process(qvs)\Working Set

    \Process(qvb)\% Processor Time

    \Process(qvb)\Private Bytes

    \Process(qvb)\Thread Count

    \Process(qvb)\Virtual Bytes

    \Process(qvb)\Working Set

    \Process(w3wp)\Private Bytes

    \Process(w3wp)\Thread Count

    \Process(w3wp)\Virtual Bytes

    \Process(w3wp)\Working Set

    \Process(w3wp)\% Processor Time

    \Processor(_Total)\% Interrupt Time

    \Processor(_Total)\% Processor Time

    \Processor(_Total)\Interrupts/sec

    \QlikView Distribution Service\*

    \QlikView Management Service\*

    \System\Context Switches/sec

    \System\Processor Queue Length

    \Web Service(_Total)\Bytes Total/sec

    \Web Service(_Total)\Current Connections

    \Web Service(_Total)\Total Method Requests/sec

    \Web Service Cache\File Cache Hits %

    \Web Service Cache\Kernel: URI Cache Flushes

    \Web Service Cache\Kernel: URI Cache Hits %

    \Web Service Cache\Kernel: URI Cache Misses

     

    (n.b. LogicalDisk(D:) should be changed to the Drive where the QlikView QVW files are hosted. It is also likely that there will be mulitple "qvb" processes on the server so you should also collect qvb#1, qvb#2, etc.)


    EDIT: 22/08/2014 I've updated the Appendix and Attachments to include the process "qvb" which is associated with the reloading of QlikView documents - and will only exist whilst a reload is taking place. I've created a dashboard on the "QV_Utilisation_Reporting.qvw" that tracks this alongside "qvs". This is particularly important if you do not use a separate Publisher as these two processes could be competing for a limited amount of Memory.


    EDIT: 28/08/2014 Updated Metrics to include multiple "qvb" processes. Updated the corresponding Memory Dashboard to include a more accurate overview of Memory Utilised by QVS and QVB. Also added an additional dashboard that shows Opened Documents vs Memory Utilisation - particularly helpful in a clustered environment to tweak the balancing of documents.