Try clear your configurations and verify QMS memory consumes:
- Stop all services;
- rename directory c:\programdata\qliktech to c:\programdata\qliktech_old
- Start all services. All configurations will be default;
- Reconfigure your license and restart Qlikview Server;
- Take a look if memory consumes is ok.
Do you have a lot of documents set to preload? This means that as soon as the QVS is started, a lot of documents will be loaded into memory right away.
Could be that you do not have enough RAM compared to the amount of documents that you want to use, or too many users browsing a server with not enough RAM. Logs should be able to tell you this. Please contact Qlik Support to file a support case - a thorough investigation can then be done.