When I perform a Reload metadata or want to send a report, the Engine goes offline every now and then. after aborting the task, it comes back online. I can't make out much from the log. We are running on version May 2023
See the log below.
Product Assembly Name Product Version Class Name Timestamp Severity Hostname User name User ID WebEngine Request ID Connection ID Task ID Execution ID Report ID Request ID Transformation ID Message Exception Stacktrace
Qlik.NPrinting.Engine 23.20.1.0 Qlik.NPrinting.Engine.Consumer.EngineConsumer 20230622T163703.692+02:00 WARN SRV-QLIK-01 0 0 0 0 0 0 0 0 A timeout occurred while sending connection statuses to scheduler in ConnectionStatusesFeedbackLoop. ERROR: System.TimeoutException: The operation has timed out.↓↓ at RabbitMQ.Util.BlockingCell`1.WaitForValue(TimeSpan timeout)↓↓ at RabbitMQ.Client.Impl.SimpleBlockingRpcContinuation.GetReply(TimeSpan timeout)↓↓ at RabbitMQ.Client.Impl.ModelBase.ModelRpc(MethodBase method, ContentHeaderBase header, Byte[] body)↓↓ at RabbitMQ.Client.Framing.Impl.Model._Private_ExchangeDeclare(String exchange, String type, Boolean passive, Boolean durable, Boolean autoDelete, Boolean internal, Boolean nowait, IDictionary`2 arguments)↓↓ at RabbitMQ.Client.Impl.AutorecoveringModel.ExchangeDeclare(String exchange, String type, Boolean durable, Boolean autoDelete, IDictionary`2 arguments)↓↓ at EasyNetQ.RabbitAdvancedBus.<>c__DisplayClass27_0.<ExchangeDeclareAsync>b__0(IModel x)↓↓ at EasyNetQ.Producer.ClientCommandDispatcherExtensions.<>c__DisplayClass1_0.<InvokeAsync>b__0(IModel model)↓↓ at EasyNetQ.Producer.PersistentChannel.<InvokeChannelActionAsync>d__10`1.MoveNext()↓↓--- End of stack trace from previous location where exception was thrown ---↓↓ at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()↓↓ at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)↓↓ at EasyNetQ.Producer.MultiChannelClientCommandDispatcher.<InvokeAsync>d__4`1.MoveNext()↓↓--- End of stack trace from previous location where exception was thrown ---↓↓ at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()↓↓ at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)↓↓ at EasyNetQ.RabbitAdvancedBus.<ExchangeDeclareAsync>d__27.MoveNext()↓↓--- End of stack trace from previous location where exception was thrown ---↓↓ at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()↓↓ at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)↓↓ at EasyNetQ.AdvancedBusExtensions.ExchangeDeclare(IAdvancedBus bus, String name, String type, Boolean durable, Boolean autoDelete, CancellationToken cancellationToken)↓↓ at Qlik.NPrinting.Common.Messages.RabbitTopologyBase.InternalPrepareQueue(String exchangeName, String queueName, CancellationToken token, Boolean useHashAsRoutingKey, Boolean exchangeDurability, Boolean queueDurability, Boolean autoDeleteQueue, Boolean autoDeleteExchange, Int32 maxPriority, Int64 queueTtlMs) in C:\Jws\release-may2023-mcFFgbjQD\server\NPrinting\src\Common\Messages\RabbitTopologyBase.cs:line 138↓↓ at Qlik.NPrinting.Common.Messages.EngineConfigurationRabbitTopology.PrepareEngineStatusQueue(CancellationToken token) in C:\Jws\release-may2023-mcFFgbjQD\server\NPrinting\src\Common\Messages\EngineConfigurationRabbitTopology.cs:line 114↓↓ at Qlik.NPrinting.Common.Messages.EngineConfigurationRabbitTopology.SendAllocationStatusToScheduler(IMessage`1 statusMsg, Int32 timeoutMillis, CancellationToken token) in C:\Jws\release-may2023-mcFFgbjQD\server\NPrinting\src\Common\Messages\EngineConfigurationRabbitTopology.cs:line 141↓↓ at Qlik.NPrinting.Engine.Consumer.EngineConsumer.ConnectionStatusesFeedbackLoop(CancellationToken cancToken) in C:\Jws\release-may2023-mcFFgbjQD\server\NPrinting\src\Engine\Consumer\EngineConsumer.cs:line 264
Qlik.NPrinting.Engine 23.20.1.0 Qlik.NPrinting.Engine.Consumer.EngineConsumer 20230622T163708.177+02:00 WARN SRV-QLIK-01 0 0 0 0 0 0 0 0 Obsolete allocation message rejected before requesting the lock. Message Timestamp 22-6-2023 14:36:37, expiration time 22-6-2023 14:36:45, delta -23153,2337 ms
Qlik.NPrinting.Engine 23.20.1.0 Qlik.NPrinting.Engine.Consumer.EngineConsumer 20230622T163708.229+02:00 ERROR SRV-QLIK-01 0 0 0 0 0 0 0 0 Error while sending connection statuses to scheduler in ConnectionStatusesFeedbackLoop. ERROR: RabbitMQ.Client.Exceptions.OperationInterruptedException: The AMQP operation was interrupted: AMQP close-reason, initiated by Library, code=0, text='End of stream', classId=0, methodId=0, cause=System.IO.EndOfStreamException: Reached the end of the stream. Possible authentication failure.↓↓ at RabbitMQ.Client.Impl.InboundFrame.ReadFrom(Stream reader, Byte[] frameHeaderBuffer)↓↓ at RabbitMQ.Client.Framing.Impl.Connection.MainLoopIteration()↓↓ at RabbitMQ.Client.Framing.Impl.Connection.MainLoop()↓↓ at RabbitMQ.Client.Impl.SimpleBlockingRpcContinuation.GetReply(TimeSpan timeout)↓↓ at RabbitMQ.Client.Impl.ModelBase.ModelRpc(MethodBase method, ContentHeaderBase header, Byte[] body)↓↓ at RabbitMQ.Client.Framing.Impl.Model._Private_ExchangeDeclare(String exchange, String type, Boolean passive, Boolean durable, Boolean autoDelete, Boolean internal, Boolean nowait, IDictionary`2 arguments)↓↓ at RabbitMQ.Client.Impl.AutorecoveringModel.ExchangeDeclare(String exchange, String type, Boolean durable, Boolean autoDelete, IDictionary`2 arguments)↓↓ at EasyNetQ.RabbitAdvancedBus.<>c__DisplayClass27_0.<ExchangeDeclareAsync>b__0(IModel x)↓↓ at EasyNetQ.Producer.ClientCommandDispatcherExtensions.<>c__DisplayClass1_0.<InvokeAsync>b__0(IModel model)↓↓ at EasyNetQ.Producer.PersistentChannel.<InvokeChannelActionAsync>d__10`1.MoveNext()↓↓--- End of stack trace from previous location where exception was thrown ---↓↓ at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()↓↓ at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)↓↓ at EasyNetQ.Producer.MultiChannelClientCommandDispatcher.<InvokeAsync>d__4`1.MoveNext()↓↓--- End of stack trace from previous location where exception was thrown ---↓↓ at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()↓↓ at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)↓↓ at EasyNetQ.RabbitAdvancedBus.<ExchangeDeclareAsync>d__27.MoveNext()↓↓--- End of stack trace from previous location where exception was thrown ---↓↓ at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()↓↓ at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)↓↓ at EasyNetQ.AdvancedBusExtensions.ExchangeDeclare(IAdvancedBus bus, String name, String type, Boolean durable, Boolean autoDelete, CancellationToken cancellationToken)↓↓ at Qlik.NPrinting.Common.Messages.RabbitTopologyBase.InternalPrepareQueue(String exchangeName, String queueName, CancellationToken token, Boolean useHashAsRoutingKey, Boolean exchangeDurability, Boolean queueDurability, Boolean autoDeleteQueue, Boolean autoDeleteExchange, Int32 maxPriority, Int64 queueTtlMs) in C:\Jws\release-may2023-mcFFgbjQD\server\NPrinting\src\Common\Messages\RabbitTopologyBase.cs:line 138↓↓ at Qlik.NPrinting.Common.Messages.EngineConfigurationRabbitTopology.PrepareEngineStatusQueue(CancellationToken token) in C:\Jws\release-may2023-mcFFgbjQD\server\NPrinting\src\Common\Messages\EngineConfigurationRabbitTopology.cs:line 114↓↓ at Qlik.NPrinting.Common.Messages.EngineConfigurationRabbitTopology.SendAllocationStatusToScheduler(IMessage`1 statusMsg, Int32 timeoutMillis, CancellationToken token) in C:\Jws\release-may2023-mcFFgbjQD\server\NPrinting\src\Common\Messages\EngineConfigurationRabbitTopology.cs:line 141↓↓ at Qlik.NPrinting.Engine.Consumer.EngineConsumer.ConnectionStatusesFeedbackLoop(CancellationToken cancToken) in C:\Jws\release-may2023-mcFFgbjQD\server\NPrinting\src\Engine\Consumer\EngineConsumer.cs:line 264
Qlik.NPrinting.Engine 23.20.1.0 Qlik.NPrinting.Engine.Engine 20230622T163708.229+02:00 ERROR SRV-QLIK-01 0 0 0 0 0 0 0 0 Network error. ERROR: System.Threading.Tasks.TaskCanceledException: A task was canceled.↓↓ at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)↓↓ at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)↓↓ at EasyNetQ.Producer.PublishConfirmationListener.PublishPendingConfirmation.<WaitAsync>d__6.MoveNext()↓↓--- End of stack trace from previous location where exception was thrown ---↓↓ at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()↓↓ at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)↓↓ at EasyNetQ.RabbitAdvancedBus.<PublishAsync>d__20.MoveNext()↓↓--- End of stack trace from previous location where exception was thrown ---↓↓ at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()↓↓ at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)↓↓ at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)↓↓ at EasyNetQ.DefaultPubSub.<PublishAsync>d__6`1.MoveNext()↓↓--- End of stack trace from previous location where exception was thrown ---↓↓ at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()↓↓ at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)↓↓ at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)↓↓ at EasyNetQ.PubSubExtensions.Publish[T](IPubSub pubSub, T message, Action`1 configure, CancellationToken cancellationToken)↓↓ at Qlik.NPrinting.Common.Engine.RabbitEngineBus.Publish[T](T message, Int32 ttl) in C:\Jws\release-may2023-mcFFgbjQD\server\NPrinting\src\Common\Engine\RabbitEngineBus.cs:line 54↓↓ at Qlik.NPrinting.Engine.Engine.PublishMessage[T](T message, Int32 ttl) in C:\Jws\release-may2023-mcFFgbjQD\server\NPrinting\src\Engine\Engine.cs:line 298
Qlik.NPrinting.Engine 23.20.1.0 Qlik.NPrinting.Engine.Consumer.EngineConsumer 20230622T163708.230+02:00 INFO SRV-QLIK-01 0 0 0 0 0 0 0 0 Stopping Engine consumer
Qlik.NPrinting.Engine 23.20.1.0 Qlik.NPrinting.Engine.Consumer.EngineConsumer 20230622T163708.231+02:00 ERROR SRV-QLIK-01 0 0 0 0 0 0 0 0 Error while sending connection statuses to scheduler in ConnectionStatusesFeedbackLoop. ERROR: RabbitMQ.Client.Exceptions.AlreadyClosedException: Already closed: The AMQP operation was interrupted: AMQP close-reason, initiated by Library, code=0, text='End of stream', classId=0, methodId=0, cause=System.IO.EndOfStreamException: Reached the end of the stream. Possible authentication failure.↓↓ at RabbitMQ.Client.Impl.InboundFrame.ReadFrom(Stream reader, Byte[] frameHeaderBuffer)↓↓ at RabbitMQ.Client.Framing.Impl.Connection.MainLoopIteration()↓↓ at RabbitMQ.Client.Framing.Impl.Connection.MainLoop()↓↓ at RabbitMQ.Client.Impl.SessionBase.Transmit(OutgoingCommand& cmd)↓↓ at RabbitMQ.Client.Impl.ModelBase.ModelRpc(MethodBase method, ContentHeaderBase header, Byte[] body)↓↓ at RabbitMQ.Client.Framing.Impl.Model._Private_ExchangeDeclare(String exchange, String type, Boolean passive, Boolean durable, Boolean autoDelete, Boolean internal, Boolean nowait, IDictionary`2 arguments)↓↓ at RabbitMQ.Client.Impl.AutorecoveringModel.ExchangeDeclare(String exchange, String type, Boolean durable, Boolean autoDelete, IDictionary`2 arguments)↓↓ at EasyNetQ.RabbitAdvancedBus.<>c__DisplayClass27_0.<ExchangeDeclareAsync>b__0(IModel x)↓↓ at EasyNetQ.Producer.ClientCommandDispatcherExtensions.<>c__DisplayClass1_0.<InvokeAsync>b__0(IModel model)↓↓ at EasyNetQ.Producer.PersistentChannel.<InvokeChannelActionAsync>d__10`1.MoveNext()↓↓--- End of stack trace from previous location where exception was thrown ---↓↓ at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()↓↓ at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)↓↓ at EasyNetQ.Producer.MultiChannelClientCommandDispatcher.<InvokeAsync>d__4`1.MoveNext()↓↓--- End of stack trace from previous location where exception was thrown ---↓↓ at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()↓↓ at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)↓↓ at EasyNetQ.RabbitAdvancedBus.<ExchangeDeclareAsync>d__27.MoveNext()↓↓--- End of stack trace from previous location where exception was thrown ---↓↓ at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()↓↓ at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)↓↓ at EasyNetQ.AdvancedBusExtensions.ExchangeDeclare(IAdvancedBus bus, String name, String type, Boolean durable, Boolean autoDelete, CancellationToken cancellationToken)↓↓ at Qlik.NPrinting.Common.Messages.RabbitTopologyBase.InternalPrepareQueue(String exchangeName, String queueName, CancellationToken token, Boolean useHashAsRoutingKey, Boolean exchangeDurability, Boolean queueDurability, Boolean autoDeleteQueue, Boolean autoDeleteExchange, Int32 maxPriority, Int64 queueTtlMs) in C:\Jws\release-may2023-mcFFgbjQD\server\NPrinting\src\Common\Messages\RabbitTopologyBase.cs:line 138↓↓ at Qlik.NPrinting.Common.Messages.EngineConfigurationRabbitTopology.PrepareEngineStatusQueue(CancellationToken token) in C:\Jws\release-may2023-mcFFgbjQD\server\NPrinting\src\Common\Messages\EngineConfigurationRabbitTopology.cs:line 114↓↓ at Qlik.NPrinting.Common.Messages.EngineConfigurationRabbitTopology.SendAllocationStatusToScheduler(IMessage`1 statusMsg, Int32 timeoutMillis, CancellationToken token) in C:\Jws\release-may2023-mcFFgbjQD\server\NPrinting\src\Common\Messages\EngineConfigurationRabbitTopology.cs:line 141↓↓ at Qlik.NPrinting.Engine.Consumer.EngineConsumer.ConnectionStatusesFeedbackLoop(CancellationToken cancToken) in C:\Jws\release-may2023-mcFFgbjQD\server\NPrinting\src\Engine\Consumer\EngineConsumer.cs:line 264
Qlik.NPrinting.Engine 23.20.1.0 Qlik.NPrinting.Engine.Consumer.EngineConsumer 20230622T163708.332+02:00 INFO SRV-QLIK-01 0 0 0 0 0 0 0 0 Engine consumer stopped successfully
