Of objects minimized, only the caption will be "calculated". Objects not showing via conditional show will (as far as I know) not be calculated at all. Objects on a different sheet will also not be calculated (until you open that sheet).
If you do not need the object at all, you can delete it. This will reduce the base memory footprint of your application by a little, which indirectly results in slightly faster applications as it results in more memory for caching which on average results in faster (though maybe not noticeable) applications.