Unbalanced Publisher Cluster issue

[Sahal]

We have a Unbalances Publisher Cluster  with 2 NODES that we have configured via the DistributionGroupDefinition.xml  - https://help.qlik.com/en-US/qlikview/May2022/Subsystems/Server/Content/QV_Server/QlikView-Server/QVS...

Two issue questions:

1.

About a week ago we started to get an issue where "Queued tasks" did not show up in QMC anymore, rather the queued tasks was shown as "Running" and then after some time will fail. (This has nothing to do with the default value in QvbWaitTimeoutMinutes).  Any suggestion why this is happening?

2. We have added 14CPU and 14cores to our Publisher environment 1CPU per 1Core, this was recommended by the VMteam to get the best performance on our viritual environment.

We did changes in DistributionGroupDefinition.xml to suite this:

<MaxSimultaneousQvbs>13</MaxSimultaneousQvbs>
      <MaxSimultaneousReaderQvbs>20</MaxSimultaneousReaderQvbs>
      <DedicatedQvbs>1</DedicatedQvbs>
      <RunDedicatedTasksAlone>false</RunDedicatedTasksAlone>
      <GraceTimeMinutes>0</GraceTimeMinutes>

And we also change some setting in RegEdit

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\SubSystems\Windows

%SystemRoot%\system32\csrss.exe ObjectDirectory=\Windows
SharedSection=1024,20480,2048 Windows=On SubSystemType=Windows
ServerDll=basesrv,1 ServerDll=winsrv:UserServerDllInitialization,3
ServerDll=winsrv:ConServerDllInitialization,2 ProfileControl=Off
MaxRequestThreads=16

But we are never able to run 13 QVB at the same time?

Can it be that the CpuOverloadLimit and MemoryOverloadLimit (is set as default) needs to be changed? We are toping max CPU but that is just for a few seconds, but should not a task start when its under 75% anyway?

A lot of questions, might need to setup a support case, but want to ask here first.

Thanks

[Chip_Matejowsky]

Hi @Sahal,

Have you reviewed the task logs for these tasks that are going into Queued state? If yes, what are you seeing there?

Yes the CpuOverloadLimit and MemoryOverloadLimit can effect task status. Qlik Knowledgebase article CPU and Memory settings to prevent overload in QlikView Distribution Service (QDS) discusses this and how to adjust these settings.

Also suggest that you review the Qlik whitepaper Scaling QlikView Publisher as it details changes you can make to the QVDistributionService.exe.config file, along with other things, to help improve performance.

Best Regards

[Sahal]

Hi @Chip_Matejowsky 

Thank you for the support. 

The CPU/Memory link that you have provided didn't really give me an answer, it just loop back to this document QlikView Tasks are not executed as expected and ar... - Qlik Community - 1715699 that suggests that we keep the default value AS-IS. However would be great to understand, if  tasks will start running as fast as the CPU and/or Memory  drops below the set values? I'm asking because in our case the CPU and Memory might peak but just for 5 sec but the task will still be on hung state. Is it a time-limit that needs to be meet, like "CPU and Memory need to be lower then the value set for at least 10 sec before a queued task will be released"?

The task logs gives us error of:

There are no anti-virus or other resources from system that are blocking Qlik processes. 

Error LoadMeasurementStorer: An unexpected error occurred while trying to update the load balancer file.. Exception=System.Exception: Failed to open \\qlikfileshare\QlikViewStorageSource\AppData\DistributionService\LoadBalancer.xml with ReadWrite access after 1 retries. Exceptions= ||    The process cannot access the file '\\qlikfileshare\QlikViewStorageSource\AppData\DistributionService\LoadBalancer.xml' because it is being used by another process. ||    at QDSMain.Clustering.ClusterFileHandler.OpenFile(FileAccess fileAccess) 

or

Error: Failed to create QlikView Engine.. Exception=System.Runtime.InteropServices.COMException (0x80004005): Error HRESULT E_FAIL has been returned from a call to a COM component. ||    at QlikView.Global.DoLogProcessSummary() ||    at QVBWrapper.Document.CreateQVBProcess(ILogBucket logBucket, Boolean createEmptyDoc)

Scaling QlikView Publisher might be next step to read...

[Chip_Matejowsky · Accepted solution]

Hi @Sahal,

The first error you mentioned is one that I typically do see when the QDS overload protection is in effect. You can try increasing the threshold values in the QVDistributionService.exe.config files of each Publisher node.

The second error you mentioned is typically associated with no COM objects available to create a QVB engine. Basically the Windows Desktop Memory Heap setting. It seems that you did increase this, but suggest that you further increase the last value to 4096. This step is also covered in the Scaling QlikView Publisher whitepaper.

Best Regards

[Sahal]

Hi @Chip_Matejowsky 

"You can try increasing the threshold values in the QVDistributionService.exe.config files of each Publisher node."

Do you mean CpuOverloadLimit and MemoryOverloadLimit ? If YES, do you have any recommendations? 14CPU/14Cores and 128GB RAM is on each NODE, with around 150+ tasks.

[Chip_Matejowsky]

Hi @Sahal,

Yes, those are the entries in the config file I’m discussing (https://community.qlik.com/t5/Official-Support-Articles/CPU-and-Memory-settings-to-prevent-overload-...).

Typically best to experiment to find value that fits best for each specific Publisher node.

[Sahal]

Hi @Chip_Matejowsky 

Thank you, I will play around with the settings. 

However i did not get an answer to this question (if you have any):

"...I'm asking because in our case the CPU and Memory might peak but just for 5 sec but the task will still be on hung state. Is it a time-limit that needs to be meet, like "CPU and Memory need to be lower then the value set for at least 10 sec before a queued task will be released"?...."

[Chip_Matejowsky]

Sorry @Sahal I don't know the answer to that question. I do know that if the resource threshold is breached the QDS stops performing tasks until the value is again under the threshold. I'm not sure if the functionality is to immediately start processing tasks again or if there is periodic polling of system resources by QDS and then tasks are restarted after going below the threshold.

Are you implementing the best practices that are outlined in the Scaling QlikView Publisher whitepaper? That is what I typically recommend or follow when helping customers with QDS performance issues.

Best Regards