Out of memory exception on publisher everyday

[#1]

Hi All,

We are facing some issues with out qlikview setup, we need your active help on this- 

We have the setup as following

1. Publisher

2. Master

3. Slave

We have started experiencing certain issues on publisher and master after we downsized the publisher from 34 GB RAM to 17 GM.

Given the tasks we have on publisher- 17 GB RAM is still good enough memory but there is something which is causing the issues.

I am pasting all the exceptions and error on publisher and on master to help identifying the root cause of the issue

Exception on  Publisher

Exception 1

File write failed: System.IO.IOException: The process cannot access the file '\\10.xx.xxx.61\Documents\LMS_HYU.qvw' because it is being used by another process.

   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)

   at System.IO.File.Delete(String path)

   at QlikTech.FileSystem.IO.File.CopyFile(Stream i_SourceStream, String i_TargetFileName, eFSSecurityMethod i_eFSSecurityMethod, List`1 i_Permissions)

Exception 2

An unexpected error occurred while trying to read notification file, C:\ProgramData\QlikTech\DistributionService\Notification.xml. Exception=System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.

   at System.Threading.Thread.StartInternal(IPrincipal principal, StackCrawlMark& stackMark)

   at System.Threading.Thread.Start()

   at SolutionGlobal.ThreadPool.ThreadPoolThread..ctor(ThreadStart i_BeginExecutionAt, String i_ThreadName)

   at SolutionGlobal.ThreadPool.ThreadPool.CheckAndStartNewThread(ILogBucket i_LogBucket, Boolean i_Force, String i_ThreadName)

   at SolutionGlobal.ThreadPool.ThreadPool.QueueJob(ILogBucket i_LogBucket, String i_ThreadName, ThreadPriority i_ThreadPriority, Boolean i_IsInternal, Delegate i_Delegate, Object[] i_Params)

   at QDSMain.Clustering.ClusterNotification.Notify(List`1 notifications)

   at QDSMain.Clustering.NotificationFiles.NotificationFile.Read(Boolean readAllAndDoNotNotify)

Exception on Master

Exception 1

System.AggregateException: One or more errors occurred. ---> System.Exception: DSC did not respond to request.

Last exception (for http://10.xx.xxx.144:4730/DSC/Service): ANY did not respond to request.

Last exception (for http://10.xx.xxx.144:4730/DSC/Service): The request channel timed out while waiting for a reply after 00:00:30. Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeout.

   at PIX.Services.ClientSupport.ClusterBase`1.Invoke[TR](CallType callType, Func`2 func, List`1 allResults, QlikMethodBehavior methodBehavior)

   at PIX.Services.ClientSupport.ClusterBase`1.Invoke[TR](CallType callType, Func`2 func)

   at QMSBackendCore.Service.ServiceImpl.CheckResourcesForDsc(DSCResource dsc, IEnumerable`1 resourceList)

   at QMSBackendCore.Service.ServiceImpl.GetDSCResourceServiceStatus(DSCResource resource, Boolean initializing)

   at QMSBackendCore.Service.ServiceImpl.<>c__DisplayClassc.<DoGetServiceStatuses>b__8(Int32 i)

   at System.Threading.Tasks.Parallel.<>c__DisplayClassf`1.<ForWorker>b__c()

   at System.Threading.Tasks.Task.InnerInvokeWithArg(Task childTask)

   at System.Threading.Tasks.Task.<>c__DisplayClass7.<ExecuteSelfReplicating>b__6(Object )

   --- End of inner exception stack trace ---

   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)

   at System.Threading.Tasks.Task.Wait()

   at System.Threading.Tasks.Parallel.ForWorker[TLocal](Int32 fromInclusive, Int32 toExclusive, ParallelOptions parallelOptions, Action`1 body, Action`2 bodyWithState, Func`4 bodyWithLocal, Func`1 localInit, Action`1 localFinally)

   at System.Threading.Tasks.Parallel.For(Int32 fromInclusive, Int32 toExclusive, Action`1 body)

   at QMSBackendCore.Service.ServiceImpl.DoGetServiceStatuses(Boolean initializing)

---> (Inner Exception #0) System.Exception: DSC did not respond to request.

Last exception (for http://10.xx.230.144:4730/DSC/Service): ANY did not respond to request.

Last exception (for http://10.xx.xxx.144:4730/DSC/Service): The request channel timed out while waiting for a reply after 00:00:30. Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeout.

   at PIX.Services.ClientSupport.ClusterBase`1.Invoke[TR](CallType callType, Func`2 func, List`1 allResults, QlikMethodBehavior methodBehavior)

   at PIX.Services.ClientSupport.ClusterBase`1.Invoke(CallType callType, Func`2 func)

   at QMSBackendCore.Service.ServiceImpl.CheckResourcesForDsc(DSCResource dsc, IEnumerable`1 resourceList)

   at QMSBackendCore.Service.ServiceImpl.GetDSCResourceServiceStatus(DSCResource resource, Boolean initializing)

   at QMSBackendCore.Service.ServiceImpl.<>c__DisplayClassc.<DoGetServiceStatuses>b__8(Int32 i)

   at System.Threading.Tasks.Parallel.<>c__DisplayClassf`1.<ForWorker>b__c()

   at System.Threading.Tasks.Task.InnerInvokeWithArg(Task childTask)

   at System.Threading.Tasks.Task.<>c__DisplayClass7.<ExecuteSelfReplicating>b__6(Object )<---

PS: The Qlikview location is excluded from Microsoft essential antivirus scan

We are not able to figure what is locking the LMS_HYU.qvd file on master  and if the OOM exception on Publisher is the result of the same?

Please help

Thanks in advance

-Jahid

[#2]

Hi Guys,

Any pointer to resolve to issue?

thanks,

Jahid

[#3]

Just wanted to point out that the whole issue starts with one error!

If you guys can tell me if this as been a known issue to you ?

Windows successfully diagnosed a low virtual memory condition. The following programs consumed the most virtual memory: QVDirectoryServiceConnector.exe (1680) consumed 38587822080 bytes, qvb.exe (36252) consumed 1971544064 bytes, and sqlservr.exe (1276) consumed 593506304 bytes.

[Anonymous]

Hello, please see Article 000045598 in Qlik's Support Portal.

The error " System.OutOfMemoryException " seems to happen because the process is unable to find a large enough section of *contiguous* unused pages in its virtual address space to do the requested mapping (not because of the total amount of memory, but rather contiguous unused space), or because a directory or template path is no longer valid. This may cause a memory leak. If the logs do not show an error related to "output producer", than it's usually because of the former.

The error "Error: QlikView NPrinting can't find a supported reporting output producer for the file" in the NPrinting logs demonstrates that there is a directory or template path that is no longer valid, or there is a duplicate template, as these will cause memory leaks... and the customer should update the path.

If you encounter an issue where the NPrinting execution remains in a running status forever, this was a bug (OP-5692) in NPrinting, and the resolution is to upgrade to the November 2017 release or newer. In summary, the report preview execution remained in "running" status forever due to an un-handled and not logged OutOfMemoryException error in Qlik NPrinting Scheduler, which is fixed, and an NPrinting (and QlikView/Qliksense) update to the latest version should be performed.

If the QlikView services are crashing when NPrinting tasks are executed, then it's likely that your QlikView documents are extremely large, charts are maximized, etc... so QlikView Server uses a lot of machine resources and it goes down right when the request from NPrinting is sent. You should try to schedule NPrinting when the Server has more resources. NPrinting uses the QlikView Desktop APIs. If the QlikView Server can't get back to QlikView Desktop, NPrinting shows the error.

Finally, ensure that the .NET Framework is updated, as Microsoft has confirmed memory leak issues which are addressed in the latest releases of the .NET Framework.