Qlik Sense Desktop - Reload suddenly very slow

[mohamed_t]

Hi everyone,

I did a windows update yesterday on my laptop. Since this update, the reload in Qlik Sense Desktop is suddenly very slow.

Before the update, the time  reload was between 1 min 15 and 2 min. Now it's around 5 minutes.

I am using Windows 11 and have 16Go Ram.

My client is QLik Sense Desktop May 2022 Patch 5

Anyone else has already encountered a similar issue please?

Thanks ! 

[lmcsedyz]

yes @marcus_sommer , it depends on very specific scenarios. I always do big performance tests before implementation to production (lots of loads for lots of clients and different types of apps / data with different approach - simple statics for one client vs. masive benchmarks of entire country with full deep dive)

But in this case and this issue, all loads are slow, no matter the size (in comparison with previous state before update)  😞

[marcus_sommer]

I assume that not always the Sense release has changed (really the same kind of installation and/or node-distribution of the services and the same configuration ?) else also the OS release and/or the environment (changed machine, VM, domain, ... and with it possible different matters in regard to the network/storage and/or security settings)? If everything is significantly slower as before it's quite unlikely an issue with the Sense release itself.

[lmcsedyz · Accepted solution]

@mohamed_t I solved it!

What has changed:

before upgrade, qlik loaded QVD and did all operation in load statement, multi threaded if possible.

But after upgrade, are all operations processed during load by single core - load from qvd and operations proces on same thread of CPU and operated during! load of each row.

But operations in in resident tables are OK and multi threaded.

DETAILs:

I did tests - only change is the way, how qlik proces the load. Before upgrade, simple operations on fields or in where statement, where processed multi threaded and fast but much higher memory consumption.

Now almost every extra instruction makes the load of QVD megaslow.

Only working cure: Always! do "vanilla" load of QVD. Desired columns from QVD without any transformation or duplicating, creating new fields - nothing.

Only allowed statement in where is "exists".

Then, do your load from this temporary vanilla table, now all your statements and wheres run multi threaded and fast, then drop vanilla temp table. Hint: exists statement in vanilla load is very fast, but in resident load is slow (inner join much faster)! So if you have bigger source QVDs (GBs), do some kind of preload, to define key for exists statement. Then do vanilla load with "exists", all other stuff in resident load and drop vanilla table when not needed anymore.

Old processing was fast but much more RAM hungry, maybe something detecting memory size is wrong after upgrade (win or qlik or both) and qlik changes how it works to make load less memory intensive.

[marcus_sommer]

Your observed cause and the workaround you developed would mean a fundamental change in the way how Qlik processed the data-flow - without any necessity and any improvements. Therefore I have serious doubts that there is such general change within the last releases.

More likely seems to me that there is any unsuitable combination of VM + OS + BIOS + Qlik settings in regard to the available hardware. It's quite normal to expect that (on the paper) identically hardware results in equally performance and that more resources would even improve the performance but the opposite could happens if the system couldn't handle all threads properly and previous parallel processes are now sequentially executed.

So IMO the above shouldn't happened and therefore the question from above again - was anything else changed unless the Qlik release within the environment. In my experience each part of a potential upgrade should be done alone which means a tool upgrade and also an OS upgrade should be always made against identically VM respectively hardware. If this worked like expected the underlying hardware might be adjusted. And on top of all a newer business release of the application-chains might be introduced. This means this are four separated steps in a testing-environment and each one carefully checked and yes it sounds like a lot of efforts - and it is - but it means also much lesser risks of issues and afterwards efforts to find and fix the reasons behind them.   

[lmcsedyz]

Server is for years without any change... only windows updates and qlik updates... Only major change was security, but it was months ago before qlik upgrades and without any impact. Other performance indicators, tests and benchmarks are without noticeable change , storage, cpu, ram, cache...

Other operations are fine, Qlik GUI resposnse is good and fast, working with apps has no issues. Just described data load.

[marcus_sommer]

It excludes some possibilities. If I think again about your description it means accessing the RAM is fast and accessing the storage/network is slow and the last may now look like a single-threaded operation but this impression may cheat and the system would assign more threads if there were more data at the time.

So it might be helpful to try various I/O testings against the storage/network - not only with Qlik else also with other tools (for example notepad++ with opening larger csv and storing them) and Windows just by copy & paste tasks. Maybe there are now any security measures in action. Another thought goes in a new/changed way of encryption in regard to access the storage/network.

[lmcsedyz]

I did... a lot of testing, because first what I was thinking about, it was storage, BUS, network and cache issues.

for latency, speed, processing files with different size etc...

But I will not work on this server much more longer. We are OEM - so it is Qlik for our clients and we will migrate at the end of the year to Qlik Cloud (multi-tenant SaaS for OEM). So I am happy it is able to run few more months, I am also testing how this scenarios, works on cloud... A lot has changed since 2020... In few cases change of approach has impact on performance even on the cloud.

At the end of the day - everything bad could do some good

[NadiaB]

Hi @mohamed_t 

Perhaps a new process or antivirus is scanning things ? You could use Process monitor pointing to your QVD's path and Qlik Sense Desktop installation folder https://community.qlik.com/t5/Official-Support-Articles/File-Access-and-Process-Monitoring-How-to-fi...

https://community.qlik.com/t5/Official-Support-Articles/Qlik-Sense-Folder-And-Files-To-Exclude-From-...

Hope it helps!