It's not just you I find the padding annoying too, especially when different objects have different amounts of padding, and then there's also padding between objects.
Anyway, have you tried adjusting the grid spacing on your sheet. If you set it to custom you can play around with the sizing. You'll need to find the best compromise for the objects you're using. I have mine set at 26.