Irregular, and super long startup time using mashups

[vegard_bakke]

We are experiencing unacceptable long startup times when opening a mashup for the first time. As in the range 5-15 seconds.

To narrow down the scope, we have tried jMeter to test repeatedly downloading the two request taking the most time:

• /resources/assets/external/requirejs/require.js
• Open the websocket 

Most requests (50-60%) take less than 100 ms, both require.js and the websocket. Fine.

25% of the requests are in the interval [100 ms, 1000 ms]. Far from great, but ok.

The rest spread out between 1 to 5 seconds. Per request. Some requests taking as long as 10 seconds to complete.

Below is a sample of a test run today. We run 20 iterations, wait a minute, 20 new iterations, etc. (Always waiting 100 ms between each request.)

Y-axis is time of the request, in seconds.

If we zoom in on four of the runs, you better see the spread.  There is an indication that the first requests are generally slower than at the end of a test run. But this is not always the case (like test run 3 in the image below).

 

We cannot find any other pattern correlating to any other known events. Apart from that there seem to be a correlation between the time opening the websocket and the time it takes downloading the require.js.

Does anyone have any ideas on how to divide and conquer this problem? 

Do you have any idea how to test if it is: Qlik? NodeJS? The OS? The disk? The VM? The Network?

We are running Qlik Sense and Qlik QAP on Windows 2016 64-bit, 32GB RAM.

It is virtual, but with dedicated memory, disk and CPU.

The test above is removing the mashup and the data model completely from the equation.  It is just looking at opening the connection, and getting a fixed file.

What are your response times? Similar?

Any feedback appreciated!

Cheers,

Vegard 🙂

[vegard_bakke]

 I believe this is more of a common scenario than auspicable.

Many thanks,
Riccardo

Yes, I feel I have this problem more often than necessarily, too.  My gut feeling is that Qlik Sense is hyper sensitive to something in an enterprise environment setup out in the wild.  But every time I have this problem, it behaves slightly differently from the last. So I still haven't solved any of them and at the same time being confident about what exactly was the root cause.

In this particular case, we installed a fresh copy on a new virtual machine.  And it was all looking very promising.  All the random lag was gone. We closed the case with Qlik Support assuming it was "something" with the other install.

A few days later, the random lag appeared on the new server as well. 

No changes had occurred on the Qlik machine (as I know of). But the virtual windows machine was sharing hardware with lots of other virtual machines.  It is therefore very hard to control the environment. 

I would not be surprised if heavy load on other virtual machines had a dramatic effect, occasionally, on Qlik Sense when serving files.  But it could also other network devices.  The real question is, why does it affect Qlik Sense so much more than any other piece of software the customers have in the same environment.

I am not able to explain how any network inspector, antivirus, or virtual machine load could cause a full tree second delay on anything.  Three seconds is huge in this context.

[djnorton]

I'm experiencing the same issue.  Our dev server's normal CPU usage is around 10% with standard tasks and daily development, but has been >95% steady ever since we put in some mashup filters.  Have you found any settings or tweaks that help?

[Guilmachado_Deloitte]

There was some progress?