Qlik Sense Scalability Tools

[Daniel_Larsson]

This package (referred to as Qlik Sense Scalability Tools) contains a complete set of tools for easy creation, execution and analysis of load/performance tests.

This tool is now deprecated and will not receive any further updates, please use the Qlik Sense Enterprise Scalability Tools instead.

Supported versions of Qlik Sense: all 2020, all 2021, 2022-aug

Included parts are:

• Standalone application for creating and executing a simulation script
• Documentation on how to use the package
• Regression analyzer
• Benchmarking package
• App evaluator package

QlikView and Qlik Sense documents to help analyze result and log files (previously included in this package) can be found here :https://community.qlik.com/docs/DOC-15451

Troubleshooting

For help to troubleshoot connection problems, please review Appendix A of the documentation or Connection Troubleshooting Tips

Change log

v5.17.0

• Add support for Qlik Sense May 2022 release
• Add support for Qlik Sense Aug 2022 release
  

v5.16.0

• Add support for Qlik Sense Feb 2022 release
  

v5.15.0

•  Add support for Qlik Sense Nov 2021 release

v5.14.0

• Add support for Qlik Sense Aug 2021 release

(See Readme.txt for changes in earlier versions of the tool.)

Your use of Qlik Sense Scalability Tool will be subject to the same license agreement between you and Qlik for your Qlik Sense License. Qlik does not provide maintenance and support services for the Qlik Sense Scalability Tool, however please check QlikCommunity for additional information on use of these products.

[Daniel_Larsson]

Micah Sakata wrote:

• Is my best bet to overload the application with many concurrent users?

As Sebastian asks, what is the purpose? One of the easiest ways would be to have a really badly designed document combined with a lot of data, e.g. a long linked list as data model instead of a nice star model, but I fail to see what value you would get from the test as this is not the kind of document which should exist in a production environment. The best value of the test would be to use a typical document for your production environment and change multiple factors such as amount of data and concurrent users.

Micah Sakata wrote:

• I found that as scenarios end, the users end their sessions as well, so do I do an infinite iteration to keep the users concurrent?

yes the end of  a scenario disconnects the user. You can have  a concurrent user thread reconnect to the same session in subsequent iterations of the scenario by setting the flag "NewUserForEachIteration" to false. The user will however connect and disconnect at start and end of the scenario.

Micah Sakata wrote:

• What's the best arrival pattern?  I've done a bunch of tests with different arrival patterns and in analyzer, I usually see just a steady linear line in sessions.  Also, on some of the patterns like exponential and LogNormal, if I set a high Max value equal to the concurrent users value, it never comes close.

Have you set the Scheduler to "DistributedScheduler"? Arrival pattern only works with this scheduler. Depending on the values set you might not come close to the max, max will only stop you from getting over this value. If you set the iterations to anything other than 1 you will affect the curve heavily as an iteration is started instantly after the other (unless using afteriterationwaiter).  As Sebastian points out there's an app to help you predict the pattern. Also please read about arrival pattern on page 11 in the documentation.

Micah Sakata wrote:

• Does the connection type affect CPU usage?  I'm using NTLM and I read the comment earlier that this would not make a big difference in exercising the engine.

Using NTLM connection does not affect the engine CPU usage much, it will however affect the Proxy and especially the Repository CPU usage. For this reason it's recommended to use header auth.

[micahsakata]

Thanks Daniel!  Very helpful.  I was tasked with seeing if I could break the system, essentially an upper boundary test on the CPU.  I'm working with a new Sense environment so we have no apps yet.  I'm testing against a Qlik demo app called "Sales Discovery".  It only has 200K rows of data so I thought the only way to test this is with a high number of concurrent users.  As Sebastian pointed out, I'd just do this with a lot of users with indefinite iterations.

My results were not as expected.  I suspect it's because I'm using NTLM.  I'll rerun with yours and Sebastian's recommendations.  Thanks a bunch for the quick response!!!

[micahsakata]

Thanks Sebastian too!  Great stuff.  Thanks for your quick response!!!  I'll modify my scenario to run linearly and extend out the run time.

As for arrival pattern helper, I saw that it generated the scenarios but it didn't help me understand the details of each type of arrival pattern so I didn't use it.  I mostly googled each distribution type and learned about them.  I tried using an exponential distribution with a lambda of 2 and when i looked at analyzer, on the session count graph, I would have expected some sort of curve.  Instead, I still saw a straight line.  Maybe I'm not understanding how this works properly.

In any case, with your recommendation, I'll just use a linear load pattern.

[rwunderlich]

I've just tried the regression analyzer with Sense 3.2.4.  Great interface, but the Regression Analyzer does not pick up most changes. It seems to be only saving and processing total rows. qDataPages data seems to be missing. Does that sound correct?

[SebastianFabian]

Hi Rob,

No this doesn't sound correct. It is supposed to record and process qDataPages. For what types of objects do you see this error and what is the expected change between the two runs?

[rwunderlich]

Hi Sebastian,

I'll demonstrate what I'm seeing with a simple example.  One table chart with one Dim and 1 Expression. I record changing to the sheet. Then I "break" the expression -- causing the values to go to zero and record again. 

I can clearly see the different results in the regression.logs, but the Regression Analyzer reports "Objects with diff: 0".

The Advanced view flags the  "qGrandTotalRow" in gray as an insignificant diff, and does not show the detail rows at all.  I've packaged the logs etc in a zip but can't seem to attach here so I'll send to you separately. Thanks for your assistance or explanation of where I've gone wrong.

-Rob

[SebastianFabian]

Thank you for your thorough answer. It seems you have discovered a bug in the diff tool.

What you're seeing as a gray diff in the data is the summation of the fields, i.e. the top-most field shown in the table. This is something we don't currently mark as significant, but I will look into changing this, so this would most likely be a simple change that at least gives some indication that something is amiss in the table.

The other, more severe bug is that the cell data is not shown in the analyzer. The data is available in the log files, but is not successfully connected to the appropriate object when doing the diff. Fixing this involves changes throughout the entire chain, most likely also including the original log production. It is unlikely that we will have time to adjust this before the next release of the tool, so it might arrive with the next minor release instead.

Once again, thanks for bringing this up to us!

[Anonymous]

How can we set a maximum number of simulated users in the qlik sense scalability tool? Please find attached screenshots of settings. It ran the test for 290 users instead of 50, and tried to simulate 7000 users (we ran out of tokens). Thanks

[jok]

Callum,

When you have the NewUserForEachIteration, it will significantly increase the number of users used.  Uncheck this and leave your ConcurrentUsers at 50.  This will only create 50 users.

[Daniel_Larsson]

Hi,

The concurrent users settings, means what it says, i.e. how many users using the application at any time (except during rampup). You've set 50 users to concurrently be in the application, simulation an infinite amount of iterations (-1) for infinite execution time (-1) (i.e. simulating until manually stopped). As soon as a user finishes your defined scenario another user will start the scenario (depending on afteriterationwait and newuserforeachiteration settings). Please view the documentation for a description of all settings.

If you, for example, use login access type, a login access will be consumed for each new user and be valid for 1 hour for that user before another is consumed, please view the license model here: Access passes .

In other words, you can simulate an exact amount of users by setting concurrent users x iterations to the value you want. i.e to simulate 50 users you could have 10 concurrent users for 5 iterations. The time this takes to finish depends heavily on your created scenario (+afteriterationwait etc), if you also set an execution time the execution will finish on whichever finishes first, the iterations or reached execution time.

If you go the route of setting infinite iterations and limited execution time (or as you did until manually stopped), the total amount will be harder to calculate, and most heavily depend on how long time it takes a user to execute the scenario you created. Please take care to create these scenarios realistically, including adding timerdelays to simulate think time for the user, no user would ever make clicks with virtually no time between getting the results and clicking something new.