Hi,
We upgraded to QV11.2 SR12 from 11.2 SR2 a few months ago. After the upgrade, we have seen what appears to be a memory leak in the QDS process. After the service starts, everything works perfectly for about 3 days. After about 3-4 days however, we notice large backup of tasks, and eventually the service becomes completely unresponsive.
After investigating, I noticed that since the upgrade, the Private Bytes RAM of the “QVDistributionService.exe” process grows after a service restart to about 40-50GB of RAM over the course of a few days, at which point the tasks begin to see huge time delays in allocating engines to the tasks, and eventually the tasks back up due to this and the service eventually becomes completely unresponsive.
Here is an example QDS task log…
After the RAM of the QVDistributionService.exe process reaches 40GB+ (note the 12 MINUTE delay allocating an engine):
11/01/2016 13:00:12.6676566 Information Opening "C:\QlikviewData\Environments\Production\Some.qvw"
11/01/2016 13:12:20.9927573 Information Allocating new QlikView Engine. Current usage count=14 of 40 (of type non-reader).
After the service is restarted… (and this is how it behaves in the 3 days leading up to the Private Bytes hitting 40GB)
Before the RAM of the QVDistributionService.exe process reaches 40GB:
11/01/2016 13:40:49.9356345 Information Opening "C:\QlikviewData\Environments\Production\Some.qvw"
11/01/2016 13:40:49.9366346 Information Allocating new QlikView Engine. Current usage count=2 of 40 (of type non-reader).
When the issue occurs, there is still some 200GB+ of physical RAM available on this server, so it is not bound by physical RAM – it’s just as if the process chokes under the weight of the memory leak.
I have seen this other case regarding the file watcher bug in .net FW, but we are on a later build of the .net FW, so I don’t believe this is related.
Publisher Distribution Tasks Extremely Slow
I have had a ticket open with support since December, but just wanted to see if anyone else is experiencing similar issues, or has any ideas or suggestions. Our QDS reloads the documents in place (it does not distribute them - it's just a simple Reload from the first tab of the task configuration), and the QVS which is on another server is mounted to load documents from the same location (which is an SSD drive located in the QDS server).
We have tried various things suggested by support (clearing task execution history files, gathering memory statistics from the server using a 3rd party debugging tool). The latest suggestion I have from QT support is that our QDS/QVS configuration is the cause (i.e. QVS and QDS using the same folder). This strikes me as odd as we have been running this configuration since 2011 on earlier releases of QV10 and QV11 without any issues at all, so I am a little bit surprised as to why it should stop working now, especially since I can see nothing in the release notes or the documentation which prohibits having a QDS source folder and QVS mount point being the same folder. As a temporary measure, we have had to implement nightly restarts of the QDS, which keeps the symptoms of the issue at hand, but is not acceptable for us longer term.
Just wondering if anyone has any thoughts, ideas, experiences or suggestions?
Thanks and Regards,
Graeme
