QMC Tasks failing sporadically with timeout error

[Jubline]

Hi.

I'm hoping someone can suggest what may be going wrong with our Qlikview QMC tasks.

Back in March, we noticed that the tasks to reload our QVWs started to fail from time-to-time. The only change to anything prior to these failures was the databases were moved to a new server. The ODBC connections were changed (and successfully tested).

The tasks always fail with a Timeout error. Mostly on trying to save the qvw. The reloads are like this:

1. Reloads a qvw in a production processing folder.

2. Does a binary load of the qvws which are accessed on AccessPoint by the users.

No user has direct access to any of the qvws (so no locking issues that I can find). We have NPrinting, but the tasks run overnight and the issues we're experiencing with the QMC Tasks occur at all different times of the day.

Reloading the qvws manually on the server works every time. The qvws take around 6 minutes to load, the binary load is just seconds.

The CPU on the server gradually gets to 100% (the Qlikview Distribution Service is the culprit here - with sometimes 7 or 8 open when I look at Task Manager). I usually have no success trying to stop or restart the service at this point and usually end up rebooting the server.

The version is Qlikview 2018

There are 2 servers in a cluster

The qvw files are on a mapped drive held on a different server

Operating System: Windows Server 2019 Standard

Server1 has 11 CPUs

Server 2 has 7 CPUs

Reloads are limited to 4 at a time

I've tried several things:

1. Changed the timeout limit to 1 hour (it was on 6 hours)

2. Changed the reload tasks to load every 2 hours (it was 30 mins)

3. Tested that the qv admin account has access to all relevant folders

4. Ensured that the Section Access includes the qv admin account

5. Added a wait after the main qvw reaload (30 seconds) on the offchance that this may help before the binary load is started.

The main problem is that this seems to be sporadic. A qvw will load fine all day and then fail several times (sometimes concurrently, other times it will run, fail, run, fail etc).

This is a sample of the log of one of the binary loads which failed earlier today (I removed the middle part of the log to just include the start and end parts). If anyone can suggest things to try, I would appreciate some help.

Thanks in advance.

(06/05/2024 10:57:50) Information: DistributeTask[bc0170ce-6b05-4cea-9ded-b716992d265a] ("Live/App1.qvw"): TryStart - Starting task. Triggered by 'TaskCompletedTrigger'. Id:5ae7c38c-7412-4bd6-a115-91d9f992edcc. ExecID:52182c24-55fc-46a0-8575-038cc376ab44

(06/05/2024 10:57:50) Information: Entering Task Execution.

... 

(06/05/2024 10:58:04) Information: Reload finished successfully

(06/05/2024 10:58:04) Information: Start saving document

(06/05/2024 11:57:50) Information: Closing the document.

(06/05/2024 11:57:51) Error: The sourcedocument failed to save.. Exception=System.Runtime.InteropServices.COMException (0x80004005): Error HRESULT E_FAIL has been returned from a call to a COM component. || at QlikView.Doc.SaveAs(String _FileName, Int16 _Format) || at QVBWrapper.Document.Save(ILogBucket i_LogBucket, String i_SaveAsFileName)

(06/05/2024 11:57:51) Information: DistributeTask[bc0170ce-6b05-4cea-9ded-b716992d265a] ("Live/App1.qvw"): TryAbort. Triggered by 'RunTimeTrigger'. Id:0000000a-000a-000a-0a0a-161616161616. ExecID:a4709667-398c-4d53-b268-aaffbbdfd7ea

(06/05/2024 11:57:51) Information: Closed the QlikView Engine successfully. ProcessID=428304

(06/05/2024 11:57:51) Error: The task "Live/App1.qvw" failed. Exception: || QDSMain.Exceptions.TaskFailedException: Task execution failed with errors to follow. ---> QDSMain.Exceptions.ReloadFailedException: Reload failed ---> QDSMain.Exceptions.TaskTimeOutException: Task <Live/App1.qvw> has timed out || at QDSMain.AbstractReloadTask.Reload(String fileName, IExecutingTaskResult executingTaskResult, String sectionAccessUserName, String sectionAccessPassword, eReloadOptions reloadOption, String variableName, String variableValue, Boolean moniterCpuUsage) || --- End of inner exception stack trace --- || at QDSMain.AbstractReloadTask.Reload(String fileName, IExecutingTaskResult executingTaskResult, String sectionAccessUserName, String sectionAccessPassword, eReloadOptions reloadOption, String variableName, String variableValue, Boolean moniterCpuUsage) || at QDSMain.DistributeTask.PerformExecute(IExecutingTaskResult executingTaskResult) || --- End of inner exception stack trace --- || at QDSMain.DistributeTask.PerformExecute(IExecutingTaskResult executingTaskResult) || at QDSMain.Task.AbstractTask.TaskExecution(CurrentExecutionArgs args)

(06/05/2024 11:57:51) Information: Task Execute Duration=01:00:01.1283403

(06/05/2024 11:57:51) Information: Sending Alert mail

(06/05/2024 11:57:52) Information: TaskResult.status=Finished

(06/05/2024 11:57:52) Information: Notifying all triggers of new state: "FinishedWithErrors"...

(06/05/2024 11:57:52) Information: Notifying all triggers of new state: "FinishedWithErrors" - Completed

[Chip_Matejowsky · Accepted solution]

Hi @Jubline,

One scenario that I have seen regarding the timeout error here is that the file may be locked by another process, such as anti-virus application or system scanning utility. Is MS Windows Defender, or other A/V software installed on the QDS server? If yes, you can test by temporarily disabling it or ensuring that the appropriate exceptions are in place for QlikView Server/Publisher. Refer to Qlik Support article QlikView Folders and Files to Exclude from Anti-virus Scanning for details. If anti-virus software isn't installed or the exceptions are already in place, you can use MS Sys Internals' Process Monitor utility to determine which process has a file lock on the app. Refer to Qlik Support article File Access and Process Monitoring - How to Find Locked Files and the Processes Locking Them for more details.

Note that QlikView version November 2018 reached end of life over three years ago and is unsupported. It is recommended to run a supported version, such as May 2023 - 12.80 (May 2024 - 12.90 should be released this month) to ensure your environment is in a supported state.

Best Regards

[Drummer]

Thanks very much for the reply @Chip_Matejowsky - much appreciated. I'll download the Process Monitor onto the server and see if we have any locks. I'll also ask the IT admin team to check the anti-virus and firewall settings. I'll come back and let you know what we find. 🙂

[Jubline]

Hi @Chip_Matejowsky - thanks for your help. The issue was the AV Scan.