You should look at how much memory QlikView uses (in the task manager), and compare that with the availabe physical memory. QlikView needs physical memory and becomes very slow when it starts swapping. Further, you should look at the response time for the straight table. If both look OK, you have no reason to worry.
20 MB is nothing. A single chart can temporarily use a very large part - maybe half - of the physical memory, without any problems.