QlikView Server (with or without Publisher) will trigger a QlikView Engince (qvb.exe) for every reload process. There is a setting that controls how many QlikView Engines might be running at the same time.
If, for example, it's set on 2 the 3rd process that wants to run will get queued and will have to wait for a qlikview engine to become available.
What you can do is, first increase the number of "Max number of simultaneous QlikView engines for distribution" to the mininum between the number of cores -1 and 9. Next you should check your reloads and make sure there will not be more running at a single time than the setting.