QlikView Issue in Automation Task

[balajibc64]

Hi, We are using Qlik Sense and Qlik View. The tasks are setup in the Qlik View and monitoring by Qlik View QMC. Till Today morning around 6.40am our task getting successful. But from 6.50 am onwards the task getting failed, suddenly all the reloaded the job getting failed. 

How to find the issue, exact root cause and where we need to find the logs. Because this is not related to any particular task. After 6.50 am all the reload task failed. 

@Albert_Candelario 

[balajibc64]

@Albert_Candelario Thanks. But not sure we can't install into our production system. Also we can't replicate this issue again, Since it was based on business dashboard. 

[balajibc64]

@Albert_Candelario Still facing issue in few task reload. I pasted the entire logs from QV QMC for that particular job. 

(6/23/2023 7:45:23 AM) Information: DistributeTask[be42b35f-4294-4b74-8d0b-0b494314417c] ("TEST\2.QVDGen\TEST1.qvw"): TryStart - Starting task. Triggered by 'TaskCompletedTrigger'. Id:af836ac8-db3b-4263-83a9-a34f56ef7fbd. ExecID:80c1a426-c5a3-48c4-8022-19ec7d2225e9
(6/23/2023 7:45:23 AM) Information: Entering Task Execution.
(6/23/2023 7:45:23 AM) Information: ClusterID=1

(6/23/2023 7:45:23 AM) Information: QDSID=9670a2a0-ea25-7781-7c19-2a1806811c30

(6/23/2023 7:45:23 AM) Information: TaskID=be42b35f-4294-4b74-8d0b-0b494314417c

(6/23/2023 7:45:23 AM) Information: ExecID=80c1a426-c5a3-48c4-8022-19ec7d2225e9

(6/23/2023 7:45:23 AM) Information: MaxRunTime=00:20:00

(6/23/2023 7:45:23 AM) Information: MachineName=SWDOHD5SQLS0002

(6/23/2023 7:45:23 AM) Information: Task execution max attempts=1

(6/23/2023 7:45:23 AM) Information: Task execution current attempt=1

(6/23/2023 7:45:23 AM) Information: Task Dependencies are OK

(6/23/2023 7:45:23 AM) Information: The task "TEST\2.QVDGen\TEST1.qvw" is starting

(6/23/2023 7:45:23 AM) Information: Document is marked to be Reloaded with fresh data. Initializing Reload for Distribution.

(6/23/2023 7:45:23 AM) Information: Opening "\\TESTSERVER\QLIK\QlikviewPRD\SourceDocs\TEST\2.QVDGen\TEST1.qvw"

(6/23/2023 7:45:23 AM) Information: Allocating new QlikView Engine. Current usage count=4 of 8 (of type non-reader).

(6/23/2023 7:45:23 AM) Information: Max retries:5

(6/23/2023 7:45:23 AM) Information: Attempt:01

(6/23/2023 7:45:24 AM) Information: Opened the QlikView Engine successfully. ProcessID=261880

(6/23/2023 7:45:24 AM) Information: Allocated QlikView Engine successfully. Current usage count=5 of 8 (of type non-reader). Ticket number=5489.

(6/23/2023 7:45:24 AM) Information: Loading document "\\TESTSERVER\QLIK\QlikviewPRD\SourceDocs\TEST\2.QVDGen\TEST1.qvw" (0.17 Mb)

(6/23/2023 7:45:24 AM) Information: Physical FileSize=0.17 Mb. Memory Allocation Delta for this file=3.54 Mb. Available Physical Memory Before Open=28046.41 Mb. Available Physical Memory After Open=28046.59 Mb. Total Physical Memory=32767.02 Mb.

(6/23/2023 7:45:24 AM) Information: Attempted to load the document without data.

(6/23/2023 7:45:24 AM) Information: The document was loaded successfully.

(6/23/2023 7:45:24 AM) Information: Document was opened successfully

(6/23/2023 7:45:24 AM) Information: Starting reload

(6/23/2023 7:45:24 AM) Information: The Source Document is being reloaded. DocumentPath=\\TESTSERVER\QLIK\QlikviewPRD\SourceDocs\TEST\2.QVDGen\TEST1.qvw

(6/23/2023 7:45:25 AM) Information: Reloading.

(6/23/2023 7:45:26 AM) Information: Reloading..

(6/23/2023 7:45:27 AM) Information: Reloading...

(6/23/2023 7:45:28 AM) Information: Reloading....

(6/23/2023 7:45:29 AM) Information: Reloading.....

(6/23/2023 7:45:30 AM) Information: Reloading......

(6/23/2023 7:45:31 AM) Information: Reloading.......

(6/23/2023 7:45:32 AM) Information: Reloading........

(6/23/2023 7:45:33 AM) Information: Reloading.........

(6/23/2023 7:45:34 AM) Information: Reloading..........

(6/23/2023 7:45:35 AM) Information: Slow down logging. Log every <2> seconds.

(6/23/2023 7:45:35 AM) Information: Reloading

(6/23/2023 7:45:37 AM) Information: Reloading.

(6/23/2023 7:45:39 AM) Information: Reloading..

(6/23/2023 7:45:41 AM) Information: Reloading...

(6/23/2023 7:45:43 AM) Information: Reloading....

(6/23/2023 7:45:45 AM) Information: Reloading.....

(6/23/2023 7:45:47 AM) Information: Reloading......

(6/23/2023 7:45:49 AM) Information: Reloading.......

(6/23/2023 7:45:51 AM) Information: Reloading........

(6/23/2023 7:45:53 AM) Information: Reloading.........

(6/23/2023 7:45:55 AM) Information: Reloading..........

(6/23/2023 7:45:57 AM) Information: Slow down logging. Log every <10> seconds.

(6/23/2023 7:45:57 AM) Information: Reloading

(6/23/2023 7:46:07 AM) Information: Reloading.

(6/23/2023 7:46:17 AM) Information: Reloading..

(6/23/2023 7:46:27 AM) Information: Reloading...

(6/23/2023 7:46:37 AM) Information: Reloading....

(6/23/2023 7:46:47 AM) Information: Reloading.....

(6/23/2023 7:46:57 AM) Information: Reloading......

(6/23/2023 7:47:07 AM) Information: Reloading.......

(6/23/2023 7:47:17 AM) Information: Reloading........

(6/23/2023 7:47:27 AM) Information: Reloading.........

(6/23/2023 7:47:37 AM) Information: Reloading..........

(6/23/2023 7:47:47 AM) Information: Slow down logging. Log every <60> seconds.

(6/23/2023 7:47:47 AM) Information: Reloading

(6/23/2023 7:48:47 AM) Information: Reloading.

(6/23/2023 7:49:47 AM) Information: Reloading..

(6/23/2023 7:50:47 AM) Information: Reloading...

(6/23/2023 7:51:47 AM) Information: Reloading....

(6/23/2023 7:52:47 AM) Information: Reloading.....

(6/23/2023 7:53:47 AM) Information: Reloading......

(6/23/2023 7:54:47 AM) Information: Reloading.......

(6/23/2023 7:55:47 AM) Information: Reloading........

(6/23/2023 7:56:47 AM) Information: Reloading.........

(6/23/2023 7:57:48 AM) Information: Reloading..........

(6/23/2023 7:58:48 AM) Information: Slow down logging. Log every <300> seconds.

(6/23/2023 7:58:48 AM) Information: Reloading

(6/23/2023 8:03:48 AM) Information: Reloading.

(6/23/2023 8:05:24 AM) Information: DistributeTask[be42b35f-4294-4b74-8d0b-0b494314417c] ("TEST\2.QVDGen\TEST1.qvw"): TryAbort. Triggered by 'RunTimeTrigger'. Id:0000000a-000a-000a-0a0a-161616161616. ExecID:f4538244-f368-4097-a291-99244c7147b1

(6/23/2023 8:05:34 AM) Warning: The QlikView Engine is Reloading, it will be killed (Please ignore logged warnings/errors about the kill).

(6/23/2023 8:05:34 AM) Information: Killing the QlikView Engine. ProcessID=261880

(6/23/2023 8:05:34 AM) Information: Reload was aborted.

(6/23/2023 8:05:37 AM) Information: Killing a child process of the QlikView Engine. ProcessID=47628. ProcessName=QvConnect64

(6/23/2023 8:05:37 AM) Information: Successfully killed a child process of the QlikView Engine. ProcessID=261880. ProcessName=qvb

(6/23/2023 8:05:52 AM) Information: Closed the QlikView Engine successfully. ProcessID=261880

(6/23/2023 8:05:52 AM) Error: The task "TEST\2.QVDGen\TEST1.qvw" failed. Exception: || QDSMain.Exceptions.TaskFailedException: Task execution failed with errors to follow. ---> QDSMain.Exceptions.ReloadFailedException: Reload failed ---> System.Threading.ThreadAbortException: Thread was being aborted. || at System.Threading.WaitHandle.WaitOneNative(SafeHandle waitableSafeHandle, UInt32 millisecondsTimeout, Boolean hasThreadAffinity, Boolean exitContext) || at System.Threading.WaitHandle.InternalWaitOne(SafeHandle waitableSafeHandle, Int64 millisecondsTimeout, Boolean hasThreadAffinity, Boolean exitContext) || at System.Threading.WaitHandle.WaitOne(Int32 millisecondsTimeout, Boolean exitContext) || at SolutionGlobal.ThreadPool.ThreadPoolJob.SafeWaitEvent(WaitHandle eventWaitHandle) || at QDSMain.AbstractReloadTask.Reload(String fileName, IExecutingTaskResult executingTaskResult, String sectionAccessUserName, String sectionAccessPassword, eReloadOptions reloadOption, String variableName, String variableValue, Boolean moniterCpuUsage) || --- End of inner exception stack trace --- || at QDSMain.AbstractReloadTask.Reload(String fileName, IExecutingTaskResult executingTaskResult, String sectionAccessUserName, String sectionAccessPassword, eReloadOptions reloadOption, String variableName, String variableValue, Boolean moniterCpuUsage) || at QDSMain.DistributeTask.PerformExecute(IExecutingTaskResult executingTaskResult) || --- End of inner exception stack trace --- || at QDSMain.DistributeTask.PerformExecute(IExecutingTaskResult executingTaskResult) || at QDSMain.Task.AbstractTask.TaskExecution(CurrentExecutionArgs args)

(6/23/2023 8:05:52 AM) Error: Task failed with exception. Exception=The currently executing operation was aborted.

(6/23/2023 8:05:52 AM) Information: Sending Alert mail

(6/23/2023 8:05:52 AM) Information: TaskResult.status=Finished

(6/23/2023 8:05:52 AM) Information: Notifying all triggers of new state: "FinishedWithErrors"...

(6/23/2023 8:05:52 AM) Information: Notifying all triggers of new state: "FinishedWithErrors" - Completed

[Albert_Candelario]

We might need the document log as seems the faulty process is the:

(6/23/2023 8:05:37 AM) Information: Killing a child process of the QlikView Engine. ProcessID=47628. ProcessName=QvConnect64

Do you have the document log, could it be shared?

Cheers,

Albert

[balajibc64]

@Albert_Candelario Document log? I couldn't get 

[Albert_Candelario]

https://community.qlik.com/t5/Official-Support-Articles/How-To-Enable-QlikView-Document-Reload-Log/t...

Kindly review.

Cheers,

Albert

[balajibc64]

@Albert_Candelario Ok will check. 

I'm having one more query regarding this task. 

If any task belongs to other task based on success, Whether it will be impact anything or it will lock other jobs?

Eg. If task A runs, but it will run based on task B success. But task B will run based on task C success. Like that we have few reloaded task. 

[Albert_Candelario]

if a task runs on success of another and such has failed, then the takes awaiting for its success will not run.

Let me know if any doubt.

Cheers,

Albert

[balajibc64]

@Albert_Candelario @Maria_Halley Today again faced same issue. Suddenly 10 am the reload task getting failed. Then I tried to restart the server, so I stopped all the Qlik Service, but QV Distribution servicex64 alone not able to stop. Its showing like "Stopping".  also it consumes the cpu and memory. 

After that we forced to stop the service and restart the server. 

Please suggest. 

[Albert_Candelario]

@balajibc64  hence do you suspect is a resource issue?

19,2% and 403MB is not that much, from the Distribution service is also good to see the qvb processes.

But how was the whole server behaving?

After restarting the services all worked fine?

Within this reduced data is hard to have a better holistic view of what could be happening. Did you enabled the document log of the failed task?

Is always the same task that it fails or it has a random behavior ?

Cheers,

Albert

[balajibc64]

@Albert_Candelario Yes. we enable the document log. No different task fails not same task. But error is almost same.  Yes after restarted the server, its fine. 

Also last week we onboarded the MS Defender, we faced some issues previously. After that we excluded some exe files related to Qlik. Then few days its working fine but suddenly issue happened. 

Now Yesterday we offboard the MS Defender in the server and we are monitoring. How its behavior.