In my experience it isn't so much how many users / charts, as how much data is behind them. A table with 5 million rows & 5 columns is tiny compared to one with 5 million rows & 100 columns. So in many respect's its more a question of how big is your app?
Two things I would suggest, 1st jump up to 10 SR2 as soon as possible (lots of bug fixes), 2nd get a copy of the QV10ServerMonitor.qvw
How much memory one specific document uses is a bit hard to tell without testing. If you restart the QVS service and load the document with one client you'll get a good idea of it. Then when you add additional users, expect an increase of somewhere around 5-15% of the document RAM usage per additional user.
Also, QVS should use a lot of memory. As long as you don't have any problems there's no cause for concern. QVS will cache all selection up until the Lower Working Set Limit (by default 70% of available RAM at the point of the service starting). Documents getting unloaded from RAM will bring the usage down a bit, but every selection made by a user will be cached up until this point. No point in having unused RAM when it can be used to hopefully improve performance for the next user opening the document.