We actually do not have any control over it. May be Qlikview engine determines the Click/Qlik and recalculates all the values of objects by refreshing it. He does not understand what object must have touched and hence it does it for all.
What best I can recommend you is that De-attach rest of the objects while performing such activity.
This would be helpful and it will save lot of time.