I am working with QS Enterprise. I have always seen this issue with Qlik Sense.
Let's say i have 256 GM RAM in all node in a cluster setup. I have an QS application with size roughly 8 GB on the disk. Once session is created for this app, Qlik Engine reserves ~ x3 or x4 times memory for this app i.e. ~ 32 GB footprint in RAM. This again increases as user session increases.
No matter how i look at this.. seems very poor! This really limits the number of apps I can keep alive in memory!
Now I understand Qlik does data compression, symbol table, bit stuffed pointers, bit map joins etc.. etc.. optimization. Consider I have Data model with 80% random values. Still a simple program can do better memory management.. I have tested by loading the same data in memory from source without any optimization and it still takes 10 GB RAM.
what is Qlik Engine (C++ program) exactly doing that it needs this much memory? WHY ?
This is normal. An app on disk is compressed and it decompresses when read into memory. With more users more cached measures are saved into memory as well.
Qlik Servers will try to keep as much as possible in memory until a lower threshold is reached. Then Qlik will try to unload things from memory (currently not used apps etc). If it is not able to do this memory usage will continue growing.
Great question. Qlik loads the entire data model (data and relationships), chart objects, etc in memory when you open the application. This is what makes Qlik Applications fast to respond to user's interactions. This is not about just loading records into memory.
What exactly happens behind the scenes I don't know but, I know how to optimize applications. If you are interested on this matter, you can use QSDA Pro to analyze your application looking for performance issues. It is a fantastic tool and may save you lots of GB: https://easyqlik.com/
@morgankejerhag I understand every Software Product does some or the other compression to save Disk I/O. Qlik must be doing some lossless compression (LZ, Deflate or some Inhouse compression Algorithm). Still does not explains the fact that it's taking x4 memory for 80% - 90% non-repeating data set. i.e. We cannot compress this much unique data by x4 times! Thus should not be consuming this much RAM either.
