You're saying that QV Server is competing for resources on the server? Are you running other memory intensive applications on it, such as databases, other server software etc.?
You might want to take a look at the task manager while running the loads to see how much available RAM you have when running the reloads. The recommendation would be to host QVS and/or Publisher on dedicated machines.