Random Task Errors

[timothyj]

My company has been battling with setting up our Windows Server 2016 Standard (64 bit edition) server. We keep getting very random task fails where the source document fails to save. It is completely random that I can tell. It happens to small document and large documents. Some can have millions of rows, some just a 5 line table. It only happens twice a week on average. It has baffled us. We are running the same documents on Windows Server 2008 R2 Datacenter Service Pack 1 (64 bit edition) with less CPUs and memory, and have not had this issue. The task log for the one today:

(12/12/2019 1:56:49 AM) Information: Slow down logging. Log every <300> seconds.
(12/12/2019 1:56:49 AM) Information: Reloading
(12/12/2019 2:01:49 AM) Information: Reloading.
(12/12/2019 2:06:49 AM) Information: Reloading..
(12/12/2019 2:11:36 AM) Information: The Source Document reload complete. DocumentPath=C:\QLIKVIEW\APPSCRIPTS\LossRatioCalcData.qvw
(12/12/2019 2:11:36 AM) Information: Memory Allocation Delta for this file=93.70 Mb. Available Physical Memory Before Reload=12899.14 Mb. Available Physical Memory After Reload=12506.99 Mb.
(12/12/2019 2:11:36 AM) Information: The Source Document was reloaded successfully.
(12/12/2019 2:11:36 AM) Information: QlikView->Settings->Document Peferences->Generate Logfile in document is set. Enabling Log Copier.
(12/12/2019 2:11:36 AM) Information: Looking for the document log "C:\QLIKVIEW\APPSCRIPTS\LossRatioCalcData.qvw" to become available.
(12/12/2019 2:11:36 AM) Information: Found a document log. FileName=C:\QLIKVIEW\APPSCRIPTS\LossRatioCalcData.qvw.log
(12/12/2019 2:11:36 AM) Information: Writing documentLog to C:\ProgramData\QlikTech\DistributionService\1\Log\20191212\014324 - App Scripts_LossRatioCalcData.qvw\DocumentLog.txt
(12/12/2019 2:11:36 AM) Information: Reload finished successfully
(12/12/2019 2:11:36 AM) Information: Start saving document
(12/12/2019 7:43:24 AM) Information: Closing the document.
(12/12/2019 7:43:24 AM) Error: The sourcedocument failed to save.. Exception=System.Runtime.InteropServices.COMException (0x80004005): Error HRESULT E_FAIL has been returned from a call to a COM component. || at QlikView.Doc.SaveAs(String _FileName, Int16 _Format) || at QVBWrapper.Document.Save(ILogBucket i_LogBucket, String i_SaveAsFileName)
(12/12/2019 7:43:24 AM) Information: Closed the QlikView Engine successfully. ProcessID=700960
(12/12/2019 7:43:24 AM) Error: The task "App Scripts/LossRatioCalcData.qvw" failed. Exception: || QDSMain.Exceptions.TaskFailedException: Task execution failed with errors to follow. ---> QDSMain.Exceptions.ReloadFailedException: Reload failed ---> QDSMain.Exceptions.TaskTimeOutException: Task <App Scripts/LossRatioCalcData.qvw> has timed out || at QDSMain.AbstractReloadTask.Reload(String fileName, IExecutingTaskResult executingTaskResult, String sectionAccessUserName, String sectionAccessPassword, eReloadOptions reloadOption, String variableName, String variableValue, Boolean moniterCpuUsage) || --- End of inner exception stack trace --- || at QDSMain.AbstractReloadTask.Reload(String fileName, IExecutingTaskResult executingTaskResult, String sectionAccessUserName, String sectionAccessPassword, eReloadOptions reloadOption, String variableName, String variableValue, Boolean moniterCpuUsage) || at QDSMain.DistributeTask.PerformExecute(IExecutingTaskResult executingTaskResult) || --- End of inner exception stack trace --- || at QDSMain.DistributeTask.PerformExecute(IExecutingTaskResult executingTaskResult) || at QDSMain.Task.AbstractTask.TaskExecution(CurrentExecutionArgs args)
(12/12/2019 7:43:24 AM) Information: Task Execute Duration=06:00:00.6057828
(12/12/2019 7:43:24 AM) Information: TaskResult.status=Finished
(12/12/2019 7:43:24 AM) Information: Notifying all triggers of new state: "FinishedWithErrors"...
(12/12/2019 7:43:24 AM) Information: Notifying all triggers of new state: "FinishedWithErrors" - Completed

We are not using Publisher. We just reset up the system from scratch and made sure to keep everything at default settings. 

[Brett_Bleess]

The only thing at this point I think would be some sort of active scanning utility that is running on the server that is causing the issue, really the only thing left, can you check on that side of things to confirm nobody installed and started running anything like that, it is usually anti-virus, but it can also be things like disk defrag utilities too...  Microsoft Sys Internals Process Monitor might help you see something, but the catch is knowing which task is going to have issues.  There are configuration changes you can make to Process Monitor not to suck up all the resources on the server, just FYI, if you look through the Help and Google search, you should be able to find those, cannot recall if we wrote up an Article or not.  

https://support.qlik.com/articles/000005295 

That is the best one I could find, I do not think it is related to stuck temp file etc., as if that were the case, pretty sure it should be causing a failure here.  It is weird it is not timing out.  Check Windows Task Manager for QVB.exe processes and make sure you only have as many of those showing up as you have active tasks, if there are more, that will indicate a hung process, kill any of those you find, that might be it, but I am not sure.  I cannot recall in the Reload Engine if we can set a task timeout or not either, I know we can in Publisher, but look on your Task screen to see if you do see an option for how long to allow the task to run, if that is there, you can use that to automatically kill things if it is not complete within the normal window and that should kick a failure alert etc.  Sorry I do not have anything better at the moment, pretty sure you have something else environment related going on at this point, or we are hitting something in .Net Framework getting messed up, what version of that are you running, you can check in the QMC\System\About, it should be something like 4.0.30319.42000, the last 5 digits is what we really care about here, and the side issue is 42000 can be anything 4.6.x, 4.7.x or 4.8.x...  

One more thing I am not sure I mentioned previously, in Server 2016, Windows Defender is fully active by default, so that could be it too, be sure to look into that too...

Regards,
Brett

[timothyj]

Document timeout is set to 480 minutes. This timed out right at 6 hours. When I just triggered it manually, it ran in 30 seconds with no trouble and the task string continued. I will forward your suggestions to IS. Old server (QV11) is rock solid. Trying to make the new one work is just killing us. 

Thanks for your help. I will keep you updated.

Tim

Task Log from last point.

(12/23/2019 3:43:39 AM) Information: Start saving document

(12/23/2019 9:43:23 AM) Information: DistributeTask[f8edcb8d-14dd-4384-975b-1ff587508660] ("Layer 2/S657157B.AAL9553D.PAYHSTPF.qvw"): TryAbort. Triggered by 'RunTimeTrigger'. Id:0000000a-000a-000a-0a0a-161616161616. ExecID:1543c8a7-1e94-473c-88ff-aafdb50d7fb6

(12/23/2019 9:43:23 AM) Information: Closing the document.

(12/23/2019 9:43:23 AM) Error: The sourcedocument failed to save.. Exception=System.Runtime.InteropServices.COMException (0x80004005): Error HRESULT E_FAIL has been returned from a call to a COM component. || at QlikView.Doc.SaveAs(String _FileName, Int16 _Format) || at QVBWrapper.Document.Save(ILogBucket i_LogBucket, String i_SaveAsFileName)

(12/23/2019 9:43:24 AM) Information: Closed the QlikView Engine successfully. ProcessID=61484

(12/23/2019 9:43:24 AM) Error: The task "Layer 2/S657157B.AAL9553D.PAYHSTPF.qvw" failed. Exception: || QDSMain.Exceptions.TaskFailedException: Task execution failed with errors to follow. ---> QDSMain.Exceptions.ReloadFailedException: Reload failed ---> QDSMain.Exceptions.TaskTimeOutException: Task <Layer 2/S657157B.AAL9553D.PAYHSTPF.qvw> has timed out || at QDSMain.AbstractReloadTask.Reload(String fileName, IExecutingTaskResult executingTaskResult, String sectionAccessUserName, String sectionAccessPassword, eReloadOptions reloadOption, String variableName, String variableValue, Boolean moniterCpuUsage) || --- End of inner exception stack trace --- || at QDSMain.AbstractReloadTask.Reload(String fileName, IExecutingTaskResult executingTaskResult, String sectionAccessUserName, String sectionAccessPassword, eReloadOptions reloadOption, String variableName, String variableValue, Boolean moniterCpuUsage) || at QDSMain.DistributeTask.PerformExecute(IExecutingTaskResult executingTaskResult) || --- End of inner exception stack trace --- || at QDSMain.DistributeTask.PerformExecute(IExecutingTaskResult executingTaskResult) || at QDSMain.Task.AbstractTask.TaskExecution(CurrentExecutionArgs args)

(12/23/2019 9:43:24 AM) Information: Task Execute Duration=06:00:00.9035289

(12/23/2019 9:43:24 AM) Information: TaskResult.status=Finished

(12/23/2019 9:43:24 AM) Information: Notifying all triggers of new state: "FinishedWithErrors"...

(12/23/2019 9:43:24 AM) Information: Notifying all triggers of new state: "FinishedWithErrors" - Completed

[Brett_Bleess]

Ok, cool, that setting is there, so you can change the 480 to a few minutes instead such that if things hang, it will kill itself if it has not completed by that timeout, but that is only a Band Aid, need to figure out what is messing with things, and the best I have would be Windows Defender if that has not been disabled on the server...  Apologies if I did not mention that previously, but I think that is where I would be looking at this point.  That would be the one major delta between 2008 R2 and 2016 Server OS... 

Just let me know what they find, but hopefully they can confirm if Defender is active, as if so, they should be able to disable it, and I would expect it should behave like 2008 R2 at that point...  You could reverse the other changes if you want to at that point, but those will not hurt anything to leave them in place...

I am around today, then out until Thursday due to holidays, just FYI.

Regards,
Brett

[timothyj]

Will have them look at all that! Thanks again for your help! Have an awesome holiday! 

[timothyj]

Windows Defender is not active. We had Trend Micro anti-virus and Dell Secure Red Cloak software. We temporarily removed those today. Net framework is version 4.0.30319.42000.

[Brett_Bleess · Accepted solution]

Ok, the other two could definitely have been causing the issues, so will be curious to see if things stop now with those disabled.  I am assuming you will not be able to leave them that way, in which case it will be important to put exceptions in place for our file directories.

This article may be of some help with that aspect:

File/Folder Exclusions 

Regards,
Brett

[timothyj]

Yeah, that is what I thought when I saw that. I thought anti-virus had been discussed anti-virus before. I forwarded that info to my IS guys who handle the servers. Thanks again!

[timothyj]

Shocker! Everything has been running fine with those two programs removed. They have now added back the Trend Micro anti-virus with proper exceptions. We will see how this goes. Thanks!

[timothyj]

So, one night on Micro Trend gave us an error. We shut that off and put Dell Secure Red Cloak on. That ran fine, n=1. We will run a few days like that. Then maybe try to reconfigure Micro Trend.

Question, is there an anti virus that may play better with Qlikview? We are open to looking at using something else.

Thanks again!

[Brett_Bleess]

Tim, yeah, I wish I had a good answer for you on anti-virus, it generally causes us more problems than it helps, but I think the key thing is to get the exceptions in place for our file/folder locations, as that seems to work the best of anything.  If the security team still wants a full scan, what I would recommend at that point is taking down the QlikView services, then run the scan on everything and once done, then bring the QlikView services back online, only sure-fire way of which I can think to be sure you cover both sides of things.  Hopefully this helps, sorry for late replay, was out for a couple of weeks.

Regards,
Brett