I believe this is by design and your only solution is to manually delete CALs after a certain time period.
I would be interested though to know if anyone has a way of automating this task...though that might go against the whole QV licensing concept.
Edit: worth noting as well is that if the license was leased less than 24 hours ago it will be quarantined by the QV server before it can be deleted. Please do correct me if this information is inaccurate.
With Visual Studio you can use this project and change it you want. Automatic delete after 24h are possible. One you must add is converting the time, because qv-server-time is ever in greenwich time and if you delete one which is not 24h not used it will been blocked this cal.