Skip to main content
Announcements
Have questions about Qlik Connect? Join us live on April 10th, at 11 AM ET: SIGN UP NOW
cancel
Showing results for 
Search instead for 
Did you mean: 
vegard_bakke
Partner - Creator III
Partner - Creator III

Irregular, and super long startup time using mashups

We are experiencing unacceptable long startup times when opening a mashup for the first time. As in the range 5-15 seconds.

 

To narrow down the scope, we have tried jMeter to test repeatedly downloading the two request taking the most time:

  • /resources/assets/external/requirejs/require.js
  • Open the websocket 

Most requests (50-60%) take less than 100 ms, both require.js and the websocket. Fine.

25% of the requests are in the interval [100 ms, 1000 ms]. Far from great, but ok.

The rest spread out between 1 to 5 seconds. Per request. Some requests taking as long as 10 seconds to complete.

 

Below is a sample of a test run today. We run 20 iterations, wait a minute, 20 new iterations, etc. (Always waiting 100 ms between each request.)

 

Y-axis is time of the request, in seconds.

2019-01-14 - Test runs.png

 

If we zoom in on four of the runs, you better see the spread.  There is an indication that the first requests are generally slower than at the end of a test run. But this is not always the case (like test run 3 in the image below).

2019-01-14 - Test runs - Zoomed.png 

We cannot find any other pattern correlating to any other known events. Apart from that there seem to be a correlation between the time opening the websocket and the time it takes downloading the require.js.

 

 

Does anyone have any ideas on how to divide and conquer this problem? 

Do you have any idea how to test if it is: Qlik? NodeJS? The OS? The disk? The VM? The Network?

 

 

We are running Qlik Sense and Qlik QAP on Windows 2016 64-bit, 32GB RAM.

It is virtual, but with dedicated memory, disk and CPU.

The test above is removing the mashup and the data model completely from the equation.  It is just looking at opening the connection, and getting a fixed file.

 

What are your response times? Similar?

 

 

Any feedback appreciated!

 

Cheers,

Vegard 🙂

Labels (4)
12 Replies
vegard_bakke
Partner - Creator III
Partner - Creator III
Author

Has anyone experienced similar issues?
npapapicco
Partner - Contributor III
Partner - Contributor III

Same issue here with QAP, the app is loaded in 40 -50 seconds  and 10 to 15 seconds the browser is waiting idle after a call to a websocket. Once loaded response time of selections is 1 sec or less.

 

Nicola

vegard_bakke
Partner - Creator III
Partner - Creator III
Author

Does it always take this long time? Or does it vary?
How long does it take to download just the require.js file? Is it constant?


In our case, sometimes it is quite "fast" (5-6 seconds) while other times it can take 20-40 seconds.


It is difficult to debug, because it is hard to identify what is caused by the Qlik Server execution, what is caused by Qlik client side JavaScript code, and what is caused by the custom 3rd party/self made mashup JavaScript code. Are you able to separate these measures?

(Network transfer rate is also part of the equation. But no network lag should take 30 seconds transfer 3.7MB, as in our case).


My speculation is that fixed time lags can often be caused by client side frameworks, executing in the client. But the server side lags vary a lot, and materialize as occasionally long TTFB waits (time-to-first-byte).

My gut feeling says that bad virtualization may be a big contributor. Qlik server seems to be hyper sensitive to something. We moved one of our servers to a complete separate VMhost, where is was all by itself, and the response time was incredible. Really good.

The puzzle is, we moved it back to the same VMhost where the other Qlik server was. It stayed fast! But the one we didn't move was still erratic and slow. On the same hardware. This beats me.


If anyone from Qlik R&D reads this, I am happy to provide several links to public Qlik servers. Both with the problem, and without. It is not just me and Npapapicco. 🙂
vegard_bakke
Partner - Creator III
Partner - Creator III
Author

Yesterday we did this experiment, when the two virtual machines were both on the same VMhost hardware.

We downloaded require.js, and opened the websocket to the app, using jMeter. 

 

The duration jumps up and down a bit.

 

Download time for require.js, sorted by  when it was startedDownload time for require.js, sorted by when it was started

Time to open websocket, sorted by when it was startedTime to open websocket, sorted by when it was started

 

If we sort the x-axis by the duration of the test, we get a clearer picture of how many is out of 

 

 

Download time for require.js, sorted by durationDownload time for require.js, sorted by duration

 

Time to open websocket, sorted by durationTime to open websocket, sorted by duration

 

 

These tests exclude any client side rendering and JavaScript execution. It tests the server response time only.

We don't know what else is happening on the logical server. But both serveres here are running on the same VM hardware.

rzenere_avvale
Partner - Specialist II
Partner - Specialist II

Hi Vegard,

really interesting analysis.
Did you find any other detail on this issue and/or contacted the Support?

Cheers,
Riccardo

vegard_bakke
Partner - Creator III
Partner - Creator III
Author

Hi Riccardo,

We have contacted support, and they have escalated it. But we are still waiting for a meeting with the next people in line.

We have unfortunately not found the cause for this behaviour.  Or if anyone else have experienced similar issues. 

 

I have jMeter scripts if anyone are keen to test their own system.

 

rzenere_avvale
Partner - Specialist II
Partner - Specialist II

Hi Vegard,

please keep us updated on how the case goes on. I believe this is more of a common scenario than auspicable.

Many thanks,
Riccardo
e_petrasevicius21
Partner - Contributor III
Partner - Contributor III

Hi, Vergard,

Maybe you have any kind of solution provided by Qlik Support? We have similar situation and looking for any kind of solutions. 

francisco_rivas
Partner - Contributor II
Partner - Contributor II

Hi, Vergard,

Maybe you have any kind of solution provided by Qlik Support? We have similar situation too with November 2019 Patch 2.

Best Regards

Francisco