topic Re: Irregular, and super long startup time using mashups in Management & Governance

Irregular, and super long startup time using mashups

vegard_bakke — Sat, 16 Nov 2024 21:36:06 GMT

We are experiencing unacceptable long startup times when opening a mashup for the first time. As in the range 5-15 seconds.

To narrow down the scope, we have tried jMeter to test repeatedly downloading the two request taking the most time:

/resources/assets/external/requirejs/require.js
Open the websocket

Most requests (50-60%) take less than 100 ms, both require.js and the websocket. Fine.

25% of the requests are in the interval [100 ms, 1000 ms]. Far from great, but ok.

The rest spread out between 1 to 5 seconds. Per request. Some requests taking as long as 10 seconds to complete.

Below is a sample of a test run today. We run 20 iterations, wait a minute, 20 new iterations, etc. (Always waiting 100 ms between each request.)

Y-axis is time of the request, in seconds.

If we zoom in on four of the runs, you better see the spread. There is an indication that the first requests are generally slower than at the end of a test run. But this is not always the case (like test run 3 in the image below).

We cannot find any other pattern correlating to any other known events. Apart from that there seem to be a correlation between the time opening the websocket and the time it takes downloading the require.js.

Does anyone have any ideas on how to divide and conquer this problem?

Do you have any idea how to test if it is: Qlik? NodeJS? The OS? The disk? The VM? The Network?

We are running Qlik Sense and Qlik QAP on Windows 2016 64-bit, 32GB RAM.

It is virtual, but with dedicated memory, disk and CPU.

The test above is removing the mashup and the data model completely from the equation. It is just looking at opening the connection, and getting a fixed file.

What are your response times? Similar?

Any feedback appreciated!

Cheers,

Vegard 🙂

Re: Irregular, and super long startup time using mashups

vegard_bakke — Fri, 18 Jan 2019 07:57:32 GMT

Has anyone experienced similar issues?

Re: Irregular, and super long startup time using mashups

npapapicco — Tue, 05 Feb 2019 15:36:31 GMT

Same issue here with QAP, the app is loaded in 40 -50 seconds and 10 to 15 seconds the browser is waiting idle after a call to a websocket. Once loaded response time of selections is 1 sec or less.

Nicola

Re: Irregular, and super long startup time using mashups

vegard_bakke — Wed, 06 Feb 2019 08:36:05 GMT

Does it always take this long time? Or does it vary?
How long does it take to download just the require.js file? Is it constant?

In our case, sometimes it is quite "fast" (5-6 seconds) while other times it can take 20-40 seconds.

It is difficult to debug, because it is hard to identify what is caused by the Qlik Server execution, what is caused by Qlik client side JavaScript code, and what is caused by the custom 3rd party/self made mashup JavaScript code. Are you able to separate these measures?

(Network transfer rate is also part of the equation. But no network lag should take 30 seconds transfer 3.7MB, as in our case).

My speculation is that fixed time lags can often be caused by client side frameworks, executing in the client. But the server side lags vary a lot, and materialize as occasionally long TTFB waits (time-to-first-byte).

My gut feeling says that bad virtualization may be a big contributor. Qlik server seems to be hyper sensitive to something. We moved one of our servers to a complete separate VMhost, where is was all by itself, and the response time was incredible. Really good.

The puzzle is, we moved it back to the same VMhost where the other Qlik server was. It stayed fast! But the one we didn't move was still erratic and slow. On the same hardware. This beats me.

If anyone from Qlik R&D reads this, I am happy to provide several links to public Qlik servers. Both with the problem, and without. It is not just me and Npapapicco. 🙂

Re: Irregular, and super long startup time using mashups

vegard_bakke — Wed, 06 Feb 2019 09:16:04 GMT

Yesterday we did this experiment, when the two virtual machines were both on the same VMhost hardware.

We downloaded require.js, and opened the websocket to the app, using jMeter.

The duration jumps up and down a bit.

Download time for require.js, sorted by when it was started

Time to open websocket, sorted by when it was started

If we sort the x-axis by the duration of the test, we get a clearer picture of how many is out of

Download time for require.js, sorted by duration

Time to open websocket, sorted by duration

These tests exclude any client side rendering and JavaScript execution. It tests the server response time only.

We don't know what else is happening on the logical server. But both serveres here are running on the same VM hardware.

Re: Irregular, and super long startup time using mashups

rzenere_avvale — Tue, 21 May 2019 07:44:37 GMT

Hi Vegard,

really interesting analysis.
Did you find any other detail on this issue and/or contacted the Support?

Cheers,
Riccardo

Re: Irregular, and super long startup time using mashups

vegard_bakke — Tue, 21 May 2019 11:02:44 GMT

Hi Riccardo,

We have contacted support, and they have escalated it. But we are still waiting for a meeting with the next people in line.

We have unfortunately not found the cause for this behaviour. Or if anyone else have experienced similar issues.

I have jMeter scripts if anyone are keen to test their own system.

Re: Irregular, and super long startup time using mashups

rzenere_avvale — Tue, 21 May 2019 11:26:05 GMT

Hi Vegard,

please keep us updated on how the case goes on. I believe this is more of a common scenario than auspicable.

Many thanks,
Riccardo

Re: Irregular, and super long startup time using mashups

e_petrasevicius21 — Fri, 17 Jan 2020 12:31:29 GMT

Hi, Vergard,

Maybe you have any kind of solution provided by Qlik Support? We have similar situation and looking for any kind of solutions.

Re: Irregular, and super long startup time using mashups

francisco_rivas — Fri, 21 Feb 2020 22:13:37 GMT

Hi, Vergard,

Maybe you have any kind of solution provided by Qlik Support? We have similar situation too with November 2019 Patch 2.

Best Regards

Francisco

Re: Irregular, and super long startup time using mashups

vegard_bakke — Mon, 24 Feb 2020 09:12:16 GMT

I believe this is more of a common scenario than auspicable.

Many thanks,
Riccardo

Yes, I feel I have this problem more often than necessarily, too. My gut feeling is that Qlik Sense is hyper sensitive to something in an enterprise environment setup out in the wild. But every time I have this problem, it behaves slightly differently from the last. So I still haven't solved any of them and at the same time being confident about what exactly was the root cause.

In this particular case, we installed a fresh copy on a new virtual machine. And it was all looking very promising. All the random lag was gone. We closed the case with Qlik Support assuming it was "something" with the other install.

A few days later, the random lag appeared on the new server as well.

No changes had occurred on the Qlik machine (as I know of). But the virtual windows machine was sharing hardware with lots of other virtual machines. It is therefore very hard to control the environment.

I would not be surprised if heavy load on other virtual machines had a dramatic effect, occasionally, on Qlik Sense when serving files. But it could also other network devices. The real question is, why does it affect Qlik Sense so much more than any other piece of software the customers have in the same environment.

I am not able to explain how any network inspector, antivirus, or virtual machine load could cause a full tree second delay on anything. Three seconds is huge in this context.

Re: Irregular, and super long startup time using mashups

djnorton — Fri, 02 Oct 2020 08:10:54 GMT

I'm experiencing the same issue. Our dev server's normal CPU usage is around 10% with standard tasks and daily development, but has been >95% steady ever since we put in some mashup filters. Have you found any settings or tweaks that help?

Re: Irregular, and super long startup time using mashups

Guilmachado_Deloitte — Wed, 04 Oct 2023 19:06:31 GMT

There was some progress?