Governance Dashboard – Performance Problems

[#1]

Hello everybody,

I installed the Governance Dashboard on one of our QV-Servers and changed the config to MultiCluster. The Problem is that the creation task run and run and run ... (several days...) is that normal?!?

#	Server1	Server2	Server3	Server4	Server5	Server6	Server6	Server8	Server9	Server10	Server11
App.-Paths (Count)	5	5	5	5	5	5	5	5	5	5	5
Logfile-Path	Cluster1	Cluster2	Cluster3	Cluster3	Cluster4	Cluster5	Cluster5	Cluster5	Cluster5	Cluster5	Cluster5
Audit (2=NO)	2	2	2	2	2	2	2	2	2	2	2
Publisher-Path	QVD1	QVD1	QVD1	QVD1	QVD1	QVD1	QVD1	QVD1	QVD1	QVD1	QVD1

If I try to start with only 2 Servers (and 10 App.) the duration is between (2 hours – 4 hours). Sometimes the task will finish successfully and sometimes it finish not successfully.. (the config is always identical..)

Does anyone have any ideas how to solve that problem..?

MultiCluster Script

// 1. How many distinct will be scanned?

SET vHowManyServers = 11;

// 2. How many Aggregators are needed?

SET vHowManyAggregators = 3;

//3. How many years of server log history is needed?

SET vHistoricalYears = 1;

SET vMultiCluster_ProfilePath_1  = 'Governance_1.1_prod\profiles\X1';

SET vMultiCluster_FilePath_1a   = '..._PROD';

SET vMultiCluster_FilePath_1b   = '..._PROD';

SET vMultiCluster_FilePath_1c   = '..._PROD';

SET vMultiCluster_FilePath_1d   = '..._PROD';

SET vMultiCluster_FilePath_1e   = '..._PROD';

SET vExcludeFiles_or_Folders_1a  = '...';

SET vExcludeFiles_or_Folders_1b  = '...';

SET vExcludeFiles_or_Folders_1c  = '...';

SET vExcludeFiles_or_Folders_1d  = '...';

SET vExcludeFiles_or_Folders_1e  = '...';

SET vMultiCluster_ServerPath_1   = '\\..\Cluster1\LogFiles';

SET vMultiCluster_HasAudit_1   = 2; (2 = NO right?)

SET vMultiCluster_PublisherPath_1  = '\\..\QlikTech\DistributionService';

SET vMultiCluster_QVPRPath_1   = '...';

SET vMultiCluster_PubDateTimeFormat_1  = 'DD.MM.CCYY HH:MI:SS';

....

// Aggregator 1

SET vMultiCluster_ProfilePath_101  = 'Governance_1.1_prod\profiles\AGG1';

SET vMultiCluster_FilePath_101a  = '$(vMultiCluster_ProfilePath_1)';

SET vMultiCluster_FilePath_101b  = '$(vMultiCluster_ProfilePath_2)';

SET vMultiCluster_FilePath_101c  = '$(vMultiCluster_ProfilePath_3)';

SET vMultiCluster_FilePath_101d  = '$(vMultiCluster_ProfilePath_4)';

SET vMultiCluster_FilePath_101e  = '$(vMultiCluster_ProfilePath_5)';

SET vClusterName_101a = 'X1';

SET vClusterName_101b = 'X2';

SET vClusterName_101c = 'X3';

SET vClusterName_101d = 'X4';

SET vClusterName_101e = 'X5';

// Aggregator 2

SET vMultiCluster_ProfilePath_102  = 'Governance_1.1_prod\profiles\AGG2';

SET vMultiCluster_FilePath_102a  = '$(vMultiCluster_ProfilePath_6)';

SET vMultiCluster_FilePath_102b  = '$(vMultiCluster_ProfilePath_7)';

SET vMultiCluster_FilePath_102c  = '$(vMultiCluster_ProfilePath_8)';

SET vMultiCluster_FilePath_102d  = '$(vMultiCluster_ProfilePath_9)';

SET vMultiCluster_FilePath_102e  = '$(vMultiCluster_ProfilePath_10)';

SET vClusterName_102a = 'X6';

SET vClusterName_102b = 'X7';

SET vClusterName_102c = 'X8';

SET vClusterName_102d = 'X9';

SET vClusterName_102e = 'X10';

// Aggregator 3

SET vMultiCluster_ProfilePath_103  = 'Governance_1.1_prod\profiles\AGG3';

SET vMultiCluster_FilePath_103a  = '$(vMultiCluster_ProfilePath_11)';

SET vClusterName_103a = 'X11';

// GLOBAL Aggregator

SET vMultiCluster_ProfilePath_201  = 'Governance_1.1_prod\profiles\GLOBAL';

SET vMultiCluster_FilePath_201a  = '$(vMultiCluster_ProfilePath_101)';

SET vMultiCluster_FilePath_201b  = '$(vMultiCluster_ProfilePath_102)';

SET vMultiCluster_FilePath_201c  = '$(vMultiCluster_ProfilePath_103)';

QVS-Logging Properties (identical on all Servers..):

Logging

Enable Session Logging

Enable Performance Logging Every 15 Minutes

Enable Event Logging

[ ] Enable Audit Logging

Event Log Verbosity

Medium

Split Files

Daily

All Log-Files are stored on the same File-Share

• (\\Server\Cluster1\LogFiles)
• (\\Server\Cluster2\LogFiles)

File-Size (QVS-Log-Files)

• For one day: ~1Mbyte (Performance_, Events_, Sessions_,)
• For 365 days: ~260 MByte (Performance_, Events_, Sessions_)
• There are still some Audit_ Log-Files available which pump up the folder size to 20,2 GByte

Trigger (Data Loading – Reproduction QVW)

• It differs from (3 Minutes to once a day)

[Tyler_Waterfall]

This is intense!

But, it looks like you have:

55 distinct file folders

5 server clusters (5 paths where the server logs are stored)

1 Publisher (1 path where the publisher & qvpr logs are stored)

Depending on the network bandwidth and speed, I am not surprised by the performance.

However, are all 55 file paths unique? Meaning are there parents folders you could reference instead of 55 different folders?

If so, you should be able to configure it to reload 5 "servers", each of which has one of the server log paths, one of which has the publisher path, and then the file folders divided up among the five.

(For the question on Audit = ? use 0 instead of 2, though it shouldn't matter.)

[#3]

55 distinct file folders

  à All Folders are located in the same directory! e.g. (../PROD/Project1, ../PROD/Project2)

5 server clusters (5 paths where the server logs are stored)

  à 5 different paths on the same server e.g. (../Server/Cluster1/Log-Files, ../Server/Cluster2/Log-Files)

1 Publisher (1 path where the publisher & qvpr logs are stored)

  à Sorry copy/past problem in total there are 8 Publisher

Depending on the network bandwidth and speed, I am not surprised by the performance.

However, are all 55 file paths unique? Meaning are there parents folders you could reference instead of 55 different folders?

  à All projects are stored in the same directory e.g. (../PROD/Project1 ..)

If so, you should be able to configure it to reload 5 "servers", each of which has one of the server log paths, one of which has the publisher path, and then the file folders divided up among the five.

  à I think this is not possible because we have 8 Publisher in total

(For the question on Audit = ? use 0 instead of 2, though it shouldn't matter.)

  à Documentation says 2 = NO // 1 = yes?!?

What do you think is there any connection between the QVS Logging time (every 15 min) and the Triggers which start every 3 minutes / once a day??

[Tyler_Waterfall]

Ah, okay. With 8 Publisher machines you would need 8 "servers" configured in the multicluster config file.

Sorry about the audit = 2 documentation. It's been a while since I reviewed the documentation I wrote

You can mix/match the config, but it will still be a bit complicated (and long-running!).

I will invite you to the Governance Dashboard 2.0 limited beta program which will start next week. you might have better luck trying that approach rather than using Governance Dashboard 1.1.1 and multi-cluster config.

[#5]

The invitation for the Governance Dashboard 2.0 sounds nice. So I'm pleased to hearing from you.

Hopefully this new Version will produce better results, and speed up the whole application.

[#6]

I'm still waiting for any kind of response ;-(

[Tyler_Waterfall]

Governance Dashboard 2.0 Beta is open to the masses! Enjoy!