QVD optimized load benchmark

[#1]

What are your IO-benchmarks when running an optimized QVD load?

We have a new shiny server on a fat RAID. It shows a throughput of a few hundred MByte/sec when reading/writing files. An optimized load of a QVD on same server gets to a 10-30MByte/sec max. Only one CPU core is fully used and the others are idle, or the load jumps between the cores.

I wonder what might be the bottleneck? Is it the intrinsically single-threaded nature of the QVD load process or something else on that server?

Could this be some setting or an antivirus that limits this to a single core?

[stevedark]

Someone like Henric Cronström would be best placed to answer about the core usage.

You may find that IO is relatively low on the QVD load as the file is so well compressed it doesn't take much disk space (and therefore IO requirement).  The work is around indexing and associating in RAM once the file is loaded.

Steve

[hic]

My guess is that the bottleneck is disc access (although you say that throughput could be a few hundred MB/s but is only 10-30MB/s). Do you get the same load time the first time you read the file (read from disc) as you get the second time you read it (read from OS cache)?

HIC

[lse]

Hi,

This is my view on this topic.

Reading a stream of qvd data is not a cpu intensive operation. It is in fact single threaded, unless the script performs  data transformations.

For QlikView a reload will only read one file/stream at the time per qv.exe or qvb.exe (Publisher reload engine).

Many parallel instances you will put a higher load on the disk. 

The "core jumping" is, to my knowledge, under the operating systems control.

/lars

[#5]

Thanks for all the feedback. I will have to experiment to reflect on that.

But nobody has given their benchmark away. What is your highest throughput when reading in the QVD optimized files?