you need to set up the directory service connector (dsc) for group resolution to work in dms mode.
configure an active directory connection in QEMC -> system -> setup -> dsc -> active directory. normally all you need to do here is to press the little button next to the path and apply.
note the url to the dsc on the summary tab.
verify that the qvs has the same url in system -> setup -> qvs -> security.
verify that the account running the qvs is member of the local group qlikview administrators on the machine running the dsc.
restart the qvs.
In my experience, if you want to use AD groups with DMS authorization, you need to specify the DSC in Distinguished format. For example, if your domain is test.mydomain.com, then distinguished format would be: LDAP://DC=test,DC=mydomain,DC=com
You also need to be careful how you specify the AD groups in the Authorization tab since it's sensitive to .com, .local, etc. Thankfully, it's not case-sensitive :)