My company has been battling with setting up our Windows Server 2016 Standard (64 bit edition) server. We keep getting very random task fails where the source document fails to save. It is completely random that I can tell. It happens to small document and large documents. Some can have millions of rows, some just a 5 line table. It only happens twice a week on average. It has baffled us. We are running the same documents on Windows Server 2008 R2 Datacenter Service Pack 1 (64 bit edition) with less CPUs and memory, and have not had this issue. The task log for the one today:
(12/12/2019 1:56:49 AM) Information: Slow down logging. Log every <300> seconds.
(12/12/2019 1:56:49 AM) Information: Reloading
(12/12/2019 2:01:49 AM) Information: Reloading.
(12/12/2019 2:06:49 AM) Information: Reloading..
(12/12/2019 2:11:36 AM) Information: The Source Document reload complete. DocumentPath=C:\QLIKVIEW\APPSCRIPTS\LossRatioCalcData.qvw
(12/12/2019 2:11:36 AM) Information: Memory Allocation Delta for this file=93.70 Mb. Available Physical Memory Before Reload=12899.14 Mb. Available Physical Memory After Reload=12506.99 Mb.
(12/12/2019 2:11:36 AM) Information: The Source Document was reloaded successfully.
(12/12/2019 2:11:36 AM) Information: QlikView->Settings->Document Peferences->Generate Logfile in document is set. Enabling Log Copier.
(12/12/2019 2:11:36 AM) Information: Looking for the document log "C:\QLIKVIEW\APPSCRIPTS\LossRatioCalcData.qvw" to become available.
(12/12/2019 2:11:36 AM) Information: Found a document log. FileName=C:\QLIKVIEW\APPSCRIPTS\LossRatioCalcData.qvw.log
(12/12/2019 2:11:36 AM) Information: Writing documentLog to C:\ProgramData\QlikTech\DistributionService\1\Log\20191212\014324 - App Scripts_LossRatioCalcData.qvw\DocumentLog.txt
(12/12/2019 2:11:36 AM) Information: Reload finished successfully
(12/12/2019 2:11:36 AM) Information: Start saving document
(12/12/2019 7:43:24 AM) Information: Closing the document.
(12/12/2019 7:43:24 AM) Error: The sourcedocument failed to save.. Exception=System.Runtime.InteropServices.COMException (0x80004005): Error HRESULT E_FAIL has been returned from a call to a COM component. || at QlikView.Doc.SaveAs(String _FileName, Int16 _Format) || at QVBWrapper.Document.Save(ILogBucket i_LogBucket, String i_SaveAsFileName)
(12/12/2019 7:43:24 AM) Information: Closed the QlikView Engine successfully. ProcessID=700960
(12/12/2019 7:43:24 AM) Error: The task "App Scripts/LossRatioCalcData.qvw" failed. Exception: || QDSMain.Exceptions.TaskFailedException: Task execution failed with errors to follow. ---> QDSMain.Exceptions.ReloadFailedException: Reload failed ---> QDSMain.Exceptions.TaskTimeOutException: Task <App Scripts/LossRatioCalcData.qvw> has timed out || at QDSMain.AbstractReloadTask.Reload(String fileName, IExecutingTaskResult executingTaskResult, String sectionAccessUserName, String sectionAccessPassword, eReloadOptions reloadOption, String variableName, String variableValue, Boolean moniterCpuUsage) || --- End of inner exception stack trace --- || at QDSMain.AbstractReloadTask.Reload(String fileName, IExecutingTaskResult executingTaskResult, String sectionAccessUserName, String sectionAccessPassword, eReloadOptions reloadOption, String variableName, String variableValue, Boolean moniterCpuUsage) || at QDSMain.DistributeTask.PerformExecute(IExecutingTaskResult executingTaskResult) || --- End of inner exception stack trace --- || at QDSMain.DistributeTask.PerformExecute(IExecutingTaskResult executingTaskResult) || at QDSMain.Task.AbstractTask.TaskExecution(CurrentExecutionArgs args)
(12/12/2019 7:43:24 AM) Information: Task Execute Duration=06:00:00.6057828
(12/12/2019 7:43:24 AM) Information: TaskResult.status=Finished
(12/12/2019 7:43:24 AM) Information: Notifying all triggers of new state: "FinishedWithErrors"...
(12/12/2019 7:43:24 AM) Information: Notifying all triggers of new state: "FinishedWithErrors" - Completed
We are not using Publisher. We just reset up the system from scratch and made sure to keep everything at default settings.
Bueller is here now! 🙂 Thanks for attaching the log, very helpful, I am fairly certain the issue is your Desktop Memory Heap setting on the new server being the default value of 768mb, I would go ahead and bump this to 4096mb per the following link:
That has a PDF attached to it, go to page 9 and the DeskTop Heap Size section, that will walk you through things the rest of the way, it will require a server reboot to be picked up too! Once you do that, I believe you will be all set, sorry I am running behind, but better late than never hopefully.
Shout if you have further questions, but just had another customer last week that was hitting this, and it did the trick for them, so I am pretty sure you will be in great shape after making that change. Oh, you could check the old server to see if at some point you did change that setting there, if so, be sure to add that to your documentation for future reference! 🙂
To be clear, you suggest this even though we are not running Publisher- the main focus of the pdf. This change will help with the QVS task problem.
Thanks for answering!
Timothy, correct, this setting is in play for both the Publisher as well as the Reload Engine, which I assume since you are not running Publisher, you are just using the Reload Engine, which is the same engine, the Publisher license just adds additional features! 🙂 The underlying settings are the same, so yes, just increase that setting, but like I said, check the old server to see if you did do that there, I am almost betting that server is not going to have the default value of 768mb, but if it does, still go ahead and change things on the new server, and the reason I suggested 4096, is that space is not locked up, it is just the max things are allowed to use, so that will keep you from running into issues down the road. If you want, you could do just 2048mb, I suspect that will work too, but if you increase the number of concurrent tasks etc. later, you may need to bump that again. Hopefully this makes some better sense now. Shout if you have more questions.
Thank you so much, Bueller! 🙂 I will forward this to the guys handling our servers and see how it goes. It has not happened since, but only seems to happen once or twice a week at odd intervals on random tasks. I will update you if it happens again or doesn't for a few weeks.
It should be when you get a bunch of concurrent tasks running, just FYI, if things are pretty smooth otherwise, that would definitely fit... Not sure how many Engines you have configured in the QVS Performance tab settings for reloads, but if more than 8, that will definitely need the boost to the Desktop Heap setting if you actually get that many concurrently running... Just let me know if this does not do the trick, and you can wait to mark things as solved until you know it is actually working! 🙂
The Max Concurrent Reloads setting? We have that set at 2, which must have been the default. The old server (QV11) is running the same. Should we bump it up?
These are probably the relevant specs:
Product name QlikView Distribution Service x64
Client Build Number 12.40.20100.0
Target Platform x64
QlikView Build Info release/QV12-40/38
Computer Name VMQVD
Operating System Version Win32_OperatingSystem=@ X64 (Microsoft Windows NT 6.2.9200.0)
.NET Version 4.0.30319.42000
Physical Memory 16383Mb
Available Memory 14195Mb
CPU 0 Intel(R) Xeon(R) CPU E5-2665 0 @ 2.40GHz
CPU 1 Intel(R) Xeon(R) CPU E5-2665 0 @ 2.40GHz
CPU 2 Intel(R) Xeon(R) CPU E5-2665 0 @ 2.40GHz
CPU 3 Intel(R) Xeon(R) CPU E5-2665 0 @ 2.40GHz
Thanks again! Appreciate the help!
Try this again, lost what I had typed! 🙂 I am going to assume all the services are running on the single server just so you understand from where I am coming. The first big thing to understand is the QDS/Reload Engine and the QVS services are the two big resource consumers in the environment, so that is what you need to balance out here, and you cannot move the Reload Engine to a different server, that is only possible if you have Publisher, so we have to be sure the QDS has access to enough memory by adjusting the QVS Low Working Set Limit setting. That default value is 70%, I would probably recommend you drop that to 50% in your case, and what that will do is allow the QDS to get access to half the physical memory overnight while it is doing the reloads instead of 30% currently if things are at the default value. That may actually stop the issues you are seeing now that I have the further context on your environment, but I think I would still recommend bumping the Desktop Heap size to 2048mb as well just to be sure that does not cause you any problems.
Given the 4 cores available, I think I would keep your engines set to 2, that leaves 2 for QVS. You could go to 3 if you are not doing any reloads during the day while you have users in the applications, I want to be sure the users have enough resources to be sure their experience is good. If you could double the cores to 8 in the VM, I would say you could potentially run 4 engines at that point, which in theory would likely decrease your reload window one way or another. Some of the info in that link I gave you is still pretty relevant here, so be sure to read through that, and you can shout if you have any questions, will do my best to try to answer any you have.
The one thing you wan to be sure is NOT happening is that when the reloads are occurring, the OS is using Disk Page File excessively, we want most everything to be running in physical memory for both the QDS and QVS. In the case of the QVS, anything active on the server application-wise, has to be in physical memory, but the QDS can make use of Disk Page File, but that is going to crush performance on the server and have the users experiencing some pretty poor performance. Hopefully this is making sense. Let me know if you have more questions, I am out on Friday's FYI, but I will be in Monday and Thursday next week, so I will shout back then if I miss something later today.
There are quite a few major changes to the QDS side of things between 11.xx and 12.40 track too, but that article I referenced should cover some of that for you.
We upped the Desktop Memory Heap to 4096mb. I set the Working Set Low to 50%.
It could not save a document again this morning. A small task that usually runs and saves in 30 seconds has been trying to save for almost 5 hours now. This task has never had trouble before and I have not made any changes to it in years. This just happens randomly to random tasks at random times. It usually involves hanging up on the document save. Any ideas?
(12/23/2019 3:43:34 AM) Information: Reloading.........
(12/23/2019 3:43:35 AM) Information: Reloading..........
(12/23/2019 3:43:36 AM) Information: Slow down logging. Log every <2> seconds.
(12/23/2019 3:43:36 AM) Information: Reloading
(12/23/2019 3:43:38 AM) Information: Reloading.
(12/23/2019 3:43:39 AM) Information: The Source Document reload complete. DocumentPath=C:\QLIKVIEW\QVDBUILDERS\LAYER2\S657157B.AAL9553D.PAYHSTPF.qvw
(12/23/2019 3:43:39 AM) Information: Memory Allocation Delta for this file=29.86 Mb. Available Physical Memory Before Reload=13601.22 Mb. Available Physical Memory After Reload=13409.42 Mb.
(12/23/2019 3:43:39 AM) Information: The Source Document was reloaded successfully.
(12/23/2019 3:43:39 AM) Information: QlikView->Settings->Document Peferences->Generate Logfile in document is set. Enabling Log Copier.
(12/23/2019 3:43:39 AM) Information: Looking for the document log "C:\QLIKVIEW\QVDBUILDERS\LAYER2\S657157B.AAL9553D.PAYHSTPF.qvw" to become available.
(12/23/2019 3:43:39 AM) Information: Found a document log. FileName=C:\QLIKVIEW\QVDBUILDERS\LAYER2\S657157B.AAL9553D.PAYHSTPF.qvw.log
(12/23/2019 3:43:39 AM) Information: Writing documentLog to C:\ProgramData\QlikTech\DistributionService\1\Log\20191223\034323 - Layer 2_S657157B.AAL9553D.PAYHSTPF.qvw\DocumentLog.txt
(12/23/2019 3:43:39 AM) Information: Reload finished successfully
(12/23/2019 3:43:39 AM) Information: Start saving document