This could be related to anything from Server load to bandwidth usage.
Are your databases having extremely large queries run during refresh time - ask DBA
Is your bandwidth being used elsewhere - Ask Infrastructure
Is the CPU being over burdened on the QV Server Governance Dashboard- Check
Is the QV Server low on RAM - Governance dashboard
Is it reloading during peak usage times - Governance Dashboard