qvw file (more than 40 million rows of data) interface operation is slow

[Nora104]

The qvw file loads more than 40 million rows of data from multiple QVDs. When operating the front-end interface, such as creating a new or modifying a chart and clicking OK, the chart will always take a long time to calculate, ranging from a few seconds to more than ten seconds. Greatly affect work efficiency. Is there any way to improve and solve it? Is it a problem of too much data? How much data is more suitable for qlikview? In actual work, more than 40 million rows of data are needed, what should be done?just like the picture below.

[lironbaram · Accepted solution]

HI 
first Qlikview can handle more than 1 Billion records , 
but the server processors and CPU are important parts in the quickens of QIX engine 
also the data structure can impact performance 
and complex chart expressions or charts that displays a lot of data can hurt preformance

[Nora104]

Does the server processor mean the performance of the server? Because the enterprise account has to be opened and logged in on the server, but the data is loaded locally through QVD and can be used offline, will it still be affected by server performance? Indeed, as far as I know, our company's server is not very good.

Does CPU refer to the computer's CPU? The company's IT helped improve the computer's memory from 4G to 8G, but it didn't seem to improve much. CPU and memory seem to be two things. Does this mean that a computer with a better CPU can solve it?

Regarding the data structure, our company's data structure is very simple, Star model, I think this should not be the problem.

Complicated calculations, sometimes I use a lot of aggr formulas, this may affect it, but the tables that are not used very much will also have slow problems, I will observe more, thank you.

I think there is no way to improve the server because the cost is too high. I can only improve from the computer's CPU and formula books, what do you think?

[Nora104]

I currently use two laptops, one is Lenovo ThinkPad X280 (Intel(R) Core(TM) i5-8250U CPU @ 1.60GHZ 1.68GHZ, 8GB RAM), the other is Lenovo ThinkPad X270 (Intel(R) Core(TM) i5-6300U CPU @ 2.40GHZ 2.40GHZ, 16GB RAM).

Do you think it is enough? Which one is better? Will 16GB RAM be better than 8GB RAM? But I don’t feel much better. Do I Need a better computer?

Could you give any more specific recommendations and suggestions?

[marcus_sommer · Accepted solution]

I think your laptops aren't really suitable for a larger application. QlikView is designed to run completely within the RAM. If there isn't enough RAM available some parts will be moved to the virtual RAM which means the HD/SSD. If this happens the performance will significantly decrease because a quite high amount of the data will be continuously shifted and many of the caching-features won't work and the CPU may need to calculate the same object/view multiple times.

This means you do need not only at least so much free RAM as your uncompressed application-size else additionally n percent of it - a rule of thumb says 10% per user but by rather complex objects with aggr() functions it could be much more. From your mentioned 8GB of RAM may only really 4 GB for QlikView available - around 2 GB goes for the OS and another 2 GB are easily consumed from a parallel running browser/outlook/teams or similar tools.

Further the CPU from the laptops are quite probably not powerful enough to do all the calculations especially not if everything needs to be calculated again and again because neither these CPU's are designed for such a scenario nor the cooling of the device which then leads to a throtteling of the CPU.

Beside this there may some amount of optimization within your application - only loading the needed fields and records by keeping the number of distinct fieldvalues rather low and avoiding unneeded formatting of data and within the UI no objets with (nested) if-loops and/or aggr-constructs - a star-scheme datamodel with around 40 M of records could work with not too much delayings between any interactions on your mentioned devices but it will be already a bit sketchy.

- Marcus

[Nora104]

Thank you so much for your answer.

Then do you know how to configure a computer suitable for qlikview? Is there a recommended and reference computer model? Specific requirements for CPU and memory?

I use that computer with 16GB RAM, and the CPU often reaches 100% during operation
, but the memory will not reach 100%, showing enough.

My guess is that the CPU is not good enough and the processing speed is slow. But I don't know what kind of CPU is suitable. My computer has an i5 processor. Is the i7 processor enough?

[Brett_Bleess · Accepted solution]

The following links should prove helpful:

QIX Engine memory management 

https://help.qlik.com/en-US/qlikview/April2020/Subsystems/Client/Content/QV_QlikView/LoadData/best-p...

https://help.qlik.com/en-US/qlikview/April2020/Subsystems/Client/Content/QV_QlikView/application-per...

About the best I can offer in addition to what @marcus_sommer provided.

Regards,
Brett

[marcus_sommer · Accepted solution]

I wouldn't recommend any laptop (they don't have the performance of a pc CPU whereby the biggest bottleneck is usually the cooling) else a classical pc with at least a quad-core >= 2.4 Ghz, 16 GB RAM and a SSD storage.

Beside this you should always look on the datamodel and the UI is designed because even big servers could perform with much lesser data rather slowly if the datamodel / UI is poorly designed.

- Marcus

[marcus_sommer]

Hi Brett,

please give a note to the community-admins that since some days I didn't get any information about any participation to postings to which I contributed. So I missed the answer on my suggestion ...

- Marcus