Qlik Community

QlikView Documents

Documents for QlikView related information.

QlikView Performance Monitoring

shane_spencer
Valued Contributor

QlikView Performance Monitoring

Built upon the application I developed and published under the title Processing Windows Perfmon Logs using QlikView  I've expanded on that to specifically monitor and analyse the performance of QlikView Servers (now that I manage several QlikView Environments). It is a one stop shop for viewing the Performance of the QlikView servers (using Windows Perfmon) and also analysing the various QlikView Logs. I know the QlikView System Monitor will provide some of the same functionality but I feel my version is far more comprehensive. I recommend setting up Windows Performance Monitoring at the earliest opportunity, don't wait until you experience issues as the chance to collect valuable data will have been lost.

As a prerequisite I try to ensure that all the Windows Perfmon Logs I collect are of a uniform nature:

         Have a naming convention: servername_YYYYMMDD

         Logs are saved as 'csv' files

         Collect metrics at 5 minute intervals*

         Run from midnight (00:00.01) to midnight (23:55.01)

*After some consideration I decided that 5 minute intervals were about the right frequency.

Attachments:

1) The 'QV_Utilisation_Reporting.qvw' app is the front end application for displaying the processed QVD's. I've set it to load the past 3 months data only. There's a large number of dashboards and charts that will suddenly take shape when the processed data is loaded in to it, though some may need tweeking to best display your data.

2) The 'QV_Perfmon_processing.qvw' app processes the Windows Perfmon (csv) Logs.

3) The 'QV_Log_processing.qvw' app processes the Audit, Events, Performance, Session and Weblogs. The Audit Logs are not loaded in to the front-end ('QV_Utilisation_Reporting.qvw') app but can be viewed and manipulated using this app giving greater functionality and granuality than the raw logs.

4) The 'QV-IIS_Logs_processing.qvw' app processes the IIS Logs. To be used if IIS is used instead of QlikView Web Server.

5) The 'QV_Reload-Log_processing.qvw' processes the Reload Logs on the Publisher server and calculates the Reload & Distribution time. This may be specific to our organisation only.

6) The 'QV_TODAY_processing.qvw' processes all of the above logs for today only and saves them to files that will be over-written on each reload. This is to provide at up to the minute view of Performance by reloading on a scheduled or ad-hoc basic just before the 'QV_Utilisation_Reporting.qvw' is reloaded.

7) The 'Perfmon-QlikView.xml' file is a template which can be used to set up Perfmon collectors. The Appendix also lists the relevant metrics.

Appendix:

For QlikView Servers I recommend the following Metrics be collected

\.NET CLR Memory(QVDistributionService)\# of Pinned Objects

\LogicalDisk(D:)\Avg. Disk Bytes/Transfer

\LogicalDisk(D:)\Avg. Disk sec/Read

\LogicalDisk(D:)\Avg. Disk sec/Write

\LogicalDisk(D:)\Current Disk Queue Length

\LogicalDisk(D:)\% Free Space

\LogicalDisk(D:)\Free Megabytes

\Memory\% Committed Bytes In Use

\Memory\Available MBytes

\Memory\Page Faults/sec

\Memory\Page Reads/sec

\Memory\Pages/sec

\Memory\Pool Nonpaged Bytes

\Network Interface(*)\Bytes Total/sec

\Network Interface(*)\Current Bandwidth

\Paging File(_Total)\% Usage

\PhysicalDisk(_Total)\% Disk Time

\PhysicalDisk(_Total)\Avg. Disk Bytes/Transfer

\PhysicalDisk(_Total)\Avg. Disk Queue Length

\Process(QVDistributionService)\% Processor Time

\Process(QVDistributionService)\Private Bytes

\Process(QVDistributionService)\Thread Count

\Process(QVDistributionService)\Virtual Bytes

\Process(QVDistributionService)\Working Set

\Process(QVDirectoryServiceConnector)\% Processor Time

\Process(QVDirectoryServiceConnector)\Private Bytes

\Process(QVDirectoryServiceConnector)\Thread Count

\Process(QVDirectoryServiceConnector)\Virtual Bytes

\Process(QVDirectoryServiceConnector)\Working Set

\Process(QVManagementService)\% Processor Time

\Process(QVManagementService)\Private Bytes

\Process(QVManagementService)\Thread Count

\Process(QVManagementService)\Virtual Bytes

\Process(QVManagementService)\Working Set

\Process(qvs)\% Processor Time

\Process(qvs)\Private Bytes

\Process(qvs)\Thread Count

\Process(qvs)\Virtual Bytes

\Process(qvs)\Working Set

\Process(qvb)\% Processor Time

\Process(qvb)\Private Bytes

\Process(qvb)\Thread Count

\Process(qvb)\Virtual Bytes

\Process(qvb)\Working Set

\Process(w3wp)\Private Bytes

\Process(w3wp)\Thread Count

\Process(w3wp)\Virtual Bytes

\Process(w3wp)\Working Set

\Process(w3wp)\% Processor Time

\Processor(_Total)\% Interrupt Time

\Processor(_Total)\% Processor Time

\Processor(_Total)\Interrupts/sec

\QlikView Distribution Service\*

\QlikView Management Service\*

\System\Context Switches/sec

\System\Processor Queue Length

\Web Service(_Total)\Bytes Total/sec

\Web Service(_Total)\Current Connections

\Web Service(_Total)\Total Method Requests/sec

\Web Service Cache\File Cache Hits %

\Web Service Cache\Kernel: URI Cache Flushes

\Web Service Cache\Kernel: URI Cache Hits %

\Web Service Cache\Kernel: URI Cache Misses

(n.b. LogicalDisk(D:) should be changed to the Drive where the QlikView QVW files are hosted. It is also likely that there will be mulitple "qvb" processes on the server so you should also collect qvb#1, qvb#2, etc.)


EDIT: 22/08/2014 I've updated the Appendix and Attachments to include the process "qvb" which is associated with the reloading of QlikView documents - and will only exist whilst a reload is taking place. I've created a dashboard on the "QV_Utilisation_Reporting.qvw" that tracks this alongside "qvs". This is particularly important if you do not use a separate Publisher as these two processes could be competing for a limited amount of Memory.


EDIT: 28/08/2014 Updated Metrics to include multiple "qvb" processes. Updated the corresponding Memory Dashboard to include a more accurate overview of Memory Utilised by QVS and QVB. Also added an additional dashboard that shows Opened Documents vs Memory Utilisation - particularly helpful in a clustered environment to tweak the balancing of documents.

Labels (2)
Attachments
Comments
MVP & Luminary
MVP & Luminary

Thanks for sharing, Shane.

Not applicable

Same here Shane, thanks for sharing.  It looks like you have done all the heavy lifting.

Thanks, Tim

Excellent, Nice work.

Regards

ASHFAQ

maleksafa
Valued Contributor

Can you please tell us more on the windows performance monitoring setup? any specific Data collector sets?

0 Likes
shane_spencer
Valued Contributor

I'm not sure what you mean Malek Safa as the list of Performance Counters are listed above in the appendix and I've also attached the Perfmon-QlikView.xml file that can be used as a template (with a bit of tweaking). Furthermore I've also stated the required format, naming convention, interval and duration.

0 Likes
Not applicable

Thanks you for sharing

shane_spencer
Valued Contributor

I've created an in-depth document on How to set up Windows Performance Monitor For QlikView Servers

to compliment this post.

0 Likes
Not applicable

Hi Shane

It looks really good, however I am having abit of trouble with the QV-IIS_Logs_processing.qvw where it says:

Table not found

STORE iis_data into QV-IISLogs_201411.qvd (qvd)

(I changed the paths to paths that is valid for this company and reachable)

Any ideas?

Thanks

0 Likes
shane_spencer
Valued Contributor

Hi Carsten, there could be a number of reasons.

Do you use IIS instead of the QlikView WebServer? I guess most people won't do.

What is the log path of the IIS Log files and is this folder shared so you can scan it? Did you put full DNS name?

What is the format of the IIS Log file name? Does it start "u_ex" or just "ex"? Is the date stamp YYMMDD?

If you can send me an IIS log and show me a screen print of the log folder it would be helpful.

0 Likes
Not applicable

Hi Shane

Wow - fast reply

We use IIS.

For testing purpose (and seing how the report looks) I copied a few of the IIS files to a shared disk where they can be accesed from the server where I use Qlikview desktop (V11 SR9)

Name of one of the files is ->  u_ex141128.log

I guess it is this line that is giving issues

STORE iis_data into \\dsv.com\corp\Apps\app-093\Dev\Qlikview Storage\Private Data\Source Documents\CDA\Biztalk IIS log\Processed_$(yestermonth).qvd (qvd);

drop table iis_data;

If I leave it like you made it:

STORE iis_data into QV-IISLogs_$(yestermonth).qvd (qvd);

drop table iis_data;

Then it gives the same error

Here is 1 line from the IIS log (the smallest is 155 MB so it would be funny to send it via email:))

#Software: Microsoft Internet Information Services 7.0

#Version: 1.0

#Date: 2014-11-29 23:00:00

#Fields: date time c-ip c-port cs-username s-sitename s-computername cs-host s-ip s-port cs-method cs-uri-stem sc-status sc-win32-status sc-substatus sc-bytes cs-bytes time-taken x-session x-fullpath x-debug

2014-11-29 23:00:00 10.56.1.203 4904 - FTPSVC1 I92012 - 145.218.236.174 21 USER GRIT.APP.ESB_CLDSV 331 0 0 47 25 0 00469224-cfde-43d8-9f36-011bdb7d97c0 - -

Here is a picture of the log directoryQvlog.png

0 Likes
Version history
Revision #:
1 of 1
Last update:
‎2014-06-12 09:54 AM
Updated by: