We have been heavy users of the REST connector since it was release - the way it builds a data model from JSON data is awesome - but I have also noted something quite strange.
We have one particular app that during reload makes on the order of 3500 calls to a REST API.
On a newly rebooted server running Sense Enterprise 2.2.4 (fairly beefy one, 10+ cores, 100+ GB RAM) each call to the API takes on the order of 3-4 seconds. A full reload should thus take ca 4 hours.
But it doesn't. During the course of reloading the app, each call takes longer and longer to return.
I have timed the source system, and it provides consistent response times.
But t the end of the app reload, each call via the REST connector takes a minute or more...
To add to the weirdness, the slowness of the calls through the REST connector remains also after the reload of the app has completed.
Starting a new reload, each API call takes a minute, and gets slower and slower from there....
We have had various ideas of what would cause this, so far including
1. Large log files that take longer and longer to write to. We have tried to minimise the size of them, and it the API calls still keep getting slower and slower
2. Accumulation of REST connector log files in C:\ProgramData\Qlik\Custom Data\QvRestConnector\Log. Each call creates one EMPTY log file (why is it even created??). To rule out tens of thousands of such empty log files causing the slowness, we have a scheduled task that delete them every few hours. No effect on the increasing slowness.
3. General server load. Slow API calls occur even when this particular job is the only one running on the server.
Looking at the QvRestConnector.exe process, it seems that process is started once per API call, then it terminates, then it restarts when a new call is made. Seems unlikely that some state would be transferred to the new process instance...
Could it be some state in the Engine.exe process that gets successively larger and larger, or more and more corrupt, or...?
The only solid way we have found to reset the load time is to reboot the server... which is a really, really terrible solution..
Maybe the same reset can be achieved by restarting some combination of Sense services - needs to be tested further.
Anyone else seen this?
Ideas on what's causing it?
Edit 1:
Looks like the C:\ProgramData\Qlik\Sense\Log\Scheduler\Trace\<servername>_System_Scheduler.txt file gets a lot of writes to it... Many lines are thousands of columns wide.. Size is zero, indicating it's locked by the owning process?
Changing the "Task execution log level" in the QMC-Scheduler section to WARNING doesn't change this - or maybe a service restart is needed. Will test when current tasks have finished.
Please mark the post as a solution if it provided you with a solution to the topic at hand. Thanks!
