We are facing the same issues too. We escalated this issue to our supplier and QlikTech. QlikTech came back and asked us to upgrade to SR3, but the issue remains the same in SR3.
I have also tried various ways to make it faster such as turning off the log files, ensure all jobs are stopped and only one job is running, but still no luck.
An example, in 11.20 R2 and R3, we have a QVW which is loading QVDs. It took only 14 - 15 secs to finish the reload in Desktop client. But it took 30 - 40 secs for Publisher to reload.
I tested the same QVW in another SBE server which is running on version 10 and without publisher, it took only less than 15secs to finish the reload.
It is obviously something wrong with 11.20 Publisher!