The QVW file size is a poor indicator of the amount of RAM required on the server. If you have very large sheet objects, especially ones containing poorly associated fields from multiple tables, or with tables with no associations, Poor designs like this force QV to build a Cartesian join in memory. A Cartesion join between two tables with 10s of millions of rows in each will be truly massive and will easily exceed your RAM capacity. High cardinality fields can also have a significant impact on performance - especially composite keys built by concatenating strings..
So I would investigate the model itself to find the operations and design features which are killing your server,
Open your document without data (is available as an option in the Start Page). Navigate or create a sheet without objects, or just with list boxes. Save the document. Reload and check whether the sheet will be populated.
If you can still manipulate stuff on this sheet (meaning the document is still responding to your requests), try to figure out which sheet is reponsible for the lock-up. As soon as you found it, kill QV.exe in Task Manager and load the document again without data. Navigate to the dangerous sheet and investigate all the complex objects.