You can not use Custom User with NTFS. When you use Customer Users you are creating the users inside QV and the QlikView Server controls the security on the QVW. Also, please not that if you setup to use Custom Users you cannot user anyother Directory service. As Gary stated, NTFS is file security that is built in to Windows and you use Active Directory or local users Directory Service for that.
hey guys I know it's a very old post but I'm facing the same problem with groups in NTFS mode.
I have AD groups but when I apply the windows security on folder it seems that nothing happens.
I have posted something here: