I'm not sure you can selectively remove objects from the shared file, its a all or nothing approach i think.
If your a member of the QV Administrators group and have QMC access you should be able to disable shared objects per document, but this would be all object types including user bookmarks.
Alternatively you can restrict the creation of server objects to only specific people, so your key users are still able to use the feature.
I'd like to add to my post as you might be able to-do the automatic selective object removal with use of the PowerTools.
Server Object Handler
Enables listing, ownership changing and deletion of server objects in documents hosted on a QlikView Server remotely. Also allows ownership changing to be performed on a selected number of server objects matching a given name pattern."