A calculated dimension is an expression. If you click on an expression value then all the dimension values used to calculate that value are selected. If you click on that expression again it select the same values again. You cannot deselect these dimension values by clicking the expression value. You have to deselect the dimension values themselves.