If you mean 1 server and 5 clients, then there is only the server to take into account, as the clients will only display information sent to them by the server. Which is limited to the currently active object(s) on the currently active sheet. Nothing more.
Load times can be reduced by performing a so called Preload of heavy documents, and by performing reloads on a separate machine using a Publisher. Make sure that your server has enough RAM to keep all documents in memory and to serve your maximum set of simultaneous users.
What has to do with the preload and configure it following these steps:
See QMC-> Documents-> User Documents-> Select a doc-> Server-> Performance-> Document Control-> Preload
My server has 8 gb of ram and 4 cpus
FYI: a number of years ago, one statement by M.Robertshaw has been stamped in my brain with a glowing iron: an enterprise QlikView server should have at least 24GB of RAM and 4 to 8 cores. You may want to do something about those 8GB, because you won't run very far with documents that contain GBytes of data. That's probably a problem already in your case.
There are quite a few discussions in this community about smaller documents that when loaded cause the QVS to allocate a multiple of their expected size in cache RAM and that bump most if not all cores to 100% load. This of course depends on what expressions you defined in your document (or in the first sheet that is shown upon opening the document). The QVS may have a lot of work to do before it can display the first few objects.
It's a good thing to calculate server resource requirements using a formula or an estimate. But that's never enough. You should test your document in WebView, start from the minimum configurations mentioned above and try to deduce who's reponsible for any excesses in RAM or CPU consumption. There are no golden rules for this, just trial & error.