As a general rule, provided the development of the applications in the server is optimal, there is no need to free up any memory. QlikView is designed to work with as much memory as possible, without conflicting with the OS. That means that there are a lot of cached expressions and therefore the user experience will be better.
If you restart, you are making your users to start caching again so they are not benefitting by QlikView technology.
Check the Working Set limits in the QMC, System, Setup, QlikView Servers, Performance to make sure that the max limit leaves room enough for the operating system and other backup and antivirus software running.
Hope that makes sense.