Is it possible to identify the exact app, object or request that causes Engine.exe to exhaust RAM and restart?
Hello,
We have a recurring issue in Qlik Sense Enterprise on Windows where an Engine node runs out of memory and the Qlik Sense Engine Service restarts. This affects all connected users: sessions reconnect and the Qlik UI temporarily becomes unstable or starts flickering.
The problem is not detecting high RAM usage. We can already see:
when the node reaches the memory limit;
which applications and users were active;
which reloads and sessions were running;
the Engine.exe memory consumption before the restart.
We use tools such as Butler SOS, Operations Monitor, Log Monitor and the standard Engine logs. However, they only show what was active around the incident. They do not prove which exact application, visualization object or in-flight Engine request caused the final memory allocation.
QIX Performance / Telemetry logging provides fields such as:
ActiveUserId
AppId / DocId
ObjectId
Method
RequestId
PeakRAM / NetRAM
However, it appears that this information is written when an Engine operation finishes. If Engine.exe terminates while the offending request is still running and allocating memory, that request may never be written to the log.
The Engine healthcheck endpoint and Traffic logs may help identify what was active or which request was received last, but they still do not show the current memory allocated by each in-flight request. Therefore, this gives correlation rather than causal attribution.
What we need is something similar to:
Timestamp
Engine node
User
AppId and app name
SheetId
ObjectId
QIX method and RequestId
Current and peak memory allocated by the request
Questions:
Does Qlik Engine expose the current memory allocation of an in-flight request before that request completes?
Is there any supported API, diagnostic mode or internal Engine metric that maps memory allocation to RequestId, AppId, ObjectId and user?
Is a QIX Performance telemetry record created only after the request finishes, or can an unfinished request be captured before an Engine crash?
Has anyone built a community solution using WinDbg, ProcDump, ETW, heap tracing or another method that provides actual attribution rather than only showing active applications?
Can Hypercube, Reload and Export memory limits reliably terminate the offending request before the Engine process crashes, so that the exact AppId, ObjectId and user are recorded?
If no such mechanism exists, is this a known observability limitation of Qlik Sense Enterprise on Windows?
We are considering analyzing Engine.exe and its loaded DLLs, but before attempting any reverse engineering or unsupported instrumentation, we would like to understand whether Qlik or the community already has a solution for this problem.
Back in the QlikView server days, I worked similar problems by correlating Windows Perfmon trace data with QlikView audit logs -- sheet and field selections. I could then follow it up with some manual testing. I believe you can get similar audit events from Qlik Sense.
Atttached is the load script I used to transform perfmon data into a QVD. Then I would load that and the audit logs into a Qlik app.
Thanks — the Perfmon load script is exactly the tedious part I was about to write myself, much appreciated. Correlating it with audit data plus manual reproduction is where I've ended up too, so it's useful to hear that's still the state of the art.
Two things I've established since posting, in case they help anyone landing here:
1. Everything that polls the Engine from outside (Butler SOS metrics, healthcheck-based apps) goes blind the moment the process dies, so the final seconds are simply absent. Only what the Engine writes to local disk itself, or pushes out as it happens, survives the crash.
2. Session and audit entries are written when a session opens, not when it closes, so "who was in the room" is recoverable even when the fatal operation is not.
That second point led me to reconsider the Traffic log, which I dismissed too quickly in my original post. My understanding now is that Traffic entries are written when a request arrives, unlike telemetry records which are written on completion. If so, the request that kills the process should still be on disk — and an unmatched request (received, never answered) at the exact second the memory ramp begins would be about as close to causal attribution as this platform allows.
Can anyone confirm that Traffic logging at Debug level is written on receipt rather than completion, and that the tail survives an abrupt termination?
Worth noting for feasibility: the node that dies is a dedicated development node with three report developers on it, so production is unaffected and Debug-level Traffic volume is manageable. I appreciate this would not be an option on a production node.
