Did you resolve this? I am still looking for a solution.
I am fairly certain it's because the server has the FIPS encryption policy applied to it. I have been experimenting with a Win 2012 server and Qlik Sense 3.2. If I disable the policy and reboot, everything works. When the policy is enabled, not only can I not apply the license key, but I can't upload apps into Sense.
I toggle the policy in Group Policy under Computer configuration > Windows Settings > Security Settings > Local Policies > Security Options > "System cryptography: Use FIPS..." (My server is not part of a domain.)
Sorry, couldn't reply earlier. Hope you got this resolved.
Now when I checked on the qlik community and elsewhere, it has been advised that we have to turn off FIPS compliance check reporting.
Ways to turn off (my preference is the last one):
Levi Turner’s answer marked correct here quotes that Qlik Sense is not FIPS ready.
We got it working only when FIPS is disabled.
And am not sure how it would it work when your server is not in the domain, perhaps that would be a point of concern.