I know this question has been sitting out there for a few months, and I was just wondering if you found anything that was helpful.
I'm dealing with the very same issue in our environment. It seems that regardless of the category selected or the number of documents/thumbnails being displayed, the loading content process takes 5-10 seconds. Sometimes longer.
Like you, our environment is clustered and AP is on a different system.
One thing that we have tried is to break NTFS security inheritance on subfolders that contain qvd generator documents, which are not accessed by our users; they're only used in document reloads. The users now only see the interface documents in AP. However, the impact of this change seemed to be pretty minimal.
I've reverted to a single server, while I stabilize the environment. I'm pointing the DNS entry for the AccessPoint to primary server and speeds are significantly faster.
It might have something to do with the way we have configured the Management Services. They are running on each server, where it's recommended to centralize. The fact that we used a VM for the AccessPoint also might have caused slow downs.
We store our source and user documents in separate folder hierarchies, so the NTFS security item you mentioned wouldn't help our situation.