Qlik Community

Qlik Scalability

A group dedicated to advancing QlikView's scalability and performance. In this group we will share knowledge on how to perform load tests of Qlikview.

Highlighted
Not applicable

QVD optimized load benchmark

What are your IO-benchmarks when running an optimized QVD load?

We have a new shiny server on a fat RAID. It shows a throughput of a few hundred MByte/sec when reading/writing files. An optimized load of a QVD on same server gets to a 10-30MByte/sec max. Only one CPU core is fully used and the others are idle, or the load jumps between the cores.

I wonder what might be the bottleneck? Is it the intrinsically single-threaded nature of the QVD load process or something else on that server?

Could this be some setting or an antivirus that limits this to a single core?

Tags (1)
4 Replies
MVP
MVP

Re: QVD optimized load benchmark

Someone like Henric Cronström would be best placed to answer about the core usage.

You may find that IO is relatively low on the QVD load as the file is so well compressed it doesn't take much disk space (and therefore IO requirement).  The work is around indexing and associating in RAM once the file is loaded.

Steve

Employee
Employee

Re: QVD optimized load benchmark

My guess is that the bottleneck is disc access (although you say that throughput could be a few hundred MB/s but is only 10-30MB/s). Do you get the same load time the first time you read the file (read from disc) as you get the second time you read it (read from OS cache)?

HIC

Employee
Employee

Re: QVD optimized load benchmark

Hi,

This is my view on this topic.

Reading a stream of qvd data is not a cpu intensive operation. It is in fact single threaded, unless the script performs  data transformations.

For QlikView a reload will only read one file/stream at the time per qv.exe or qvb.exe (Publisher reload engine).

Many parallel instances you will put a higher load on the disk. 

The "core jumping" is, to my knowledge, under the operating systems control.

/lars

Not applicable

Re: QVD optimized load benchmark

Thanks for all the feedback. I will have to experiment to reflect on that.

But nobody has given their benchmark away. What is your highest throughput when reading in the QVD optimized files?