Looking at your present system I would say you do need to upgrade. One of the best resources for this is your account manager. They can work with you to find the system that will work with your environment.
Also, you need to look at server reloads and see if you can use that and move away from command line.
It's correct that the 32bit QVS can only utilize 2GB of memory. That memory must hold the loaded documents, users session states and cached results. If the QVS approaches the thresehold of 2GB, it will be forced to trim the cache. It's quite likely the random performance spikes you experience is to recalculate something that would have been in the cache but was trimmed out due to memory constraints.
Review your memory allocation in the QMC Qvs Statistics -> Performance pane. Using the default working set values, about 1.5GB VM Committed is where you'll see cache trimming. You can see the historical values and trending by looking at the server performance logs.
If you get up around 1.8GB comiitted, you may also subject QVS to paging, which is a performance killer.
Using those numbers (if my speculation is correct), you should be able to justify that moving to a 64bit OS with more RAM would improve performance.