My guess (I'm not sure 100%) is that in both cases the answer is the same. The aggregations are done by the server and the rendering is done by your computer (client). Think of it as if you see a webpage with a very big picture. The server can have a very good connection and processor and retrieve it and send it to you very quickly, but your browser may take time to display it.
So when you have a chart and you click on one values of one dimension, QlikView Server first does the required aggregations (using its memory and CPU) and "sends" you the chart with the new results. But these results may take time to be displayed in your computer.
It's worth noting that when the QVP protocol works (i. e. port 4747 is open), the Plugin connects directly to the Server, whilst AJAX will always use the Web Server (whatever it is QVWS or IIS), so the AJAX client should take longer to get the chart because there are some communications between the WebServer and the Server the browser has to wait for and that are not performed when the Plugin accesses the Server.
I do am sure about that part: all aggregations are done in the server. The ActiveX will need local resources only for its own work (if you have dozens of processes running you will render slower than if you are running only Internet Explorer.