First, set all logs in the QMC to the highest level, to gather the most information possible. Places to check within the QMC:
System, Setup, QlikView Servers, Logging, Enable Session Logging and Enable Event Logging, set the Verbosity level to Debug (to see which requests get to the QVS)
System, Setup, Directory Services Connectors, General, Debug Logging (to see how the QV is passing the credentials to the Custom DSC)
System, Setup, QlikView Web Servers, General, Logging Level set to High (to see which requests get to the AccessPoint)
This may use a lot of disk space, so test it in a very controlled environment. Make sure the users are allowed to log in according to the CUSTOM Directory and that the Directory Service Connectors are set up accordingly.