topic QVD optimized load benchmark in Scalability

QVD optimized load benchmark

Thu, 03 Apr 2014 04:13:12 GMT

What are your IO-benchmarks when running an optimized QVD load?

We have a new shiny server on a fat RAID. It shows a throughput of a few hundred MByte/sec when reading/writing files. An optimized load of a QVD on same server gets to a 10-30MByte/sec max. Only one CPU core is fully used and the others are idle, or the load jumps between the cores.

I wonder what might be the bottleneck? Is it the intrinsically single-threaded nature of the QVD load process or something else on that server?

Could this be some setting or an antivirus that limits this to a single core?

Re: QVD optimized load benchmark

stevedark — Thu, 03 Apr 2014 08:31:37 GMT

Someone like Henric Cronström would be best placed to answer about the core usage.

You may find that IO is relatively low on the QVD load as the file is so well compressed it doesn't take much disk space (and therefore IO requirement). The work is around indexing and associating in RAM once the file is loaded.

Steve

Re: QVD optimized load benchmark

hic — Thu, 03 Apr 2014 10:07:51 GMT

My guess is that the bottleneck is disc access (although you say that throughput could be a few hundred MB/s but is only 10-30MB/s). Do you get the same load time the first time you read the file (read from disc) as you get the second time you read it (read from OS cache)?

HIC

Re: QVD optimized load benchmark

lse — Thu, 03 Apr 2014 12:44:05 GMT

Hi,

This is my view on this topic.

Reading a stream of qvd data is not a cpu intensive operation. It is in fact single threaded, unless the script performs data transformations.

For QlikView a reload will only read one file/stream at the time per qv.exe or qvb.exe (Publisher reload engine).

Many parallel instances you will put a higher load on the disk.

The "core jumping" is, to my knowledge, under the operating systems control.

/lars

Re: QVD optimized load benchmark

Fri, 04 Apr 2014 06:13:03 GMT

Thanks for all the feedback. I will have to experiment to reflect on that.

But nobody has given their benchmark away. What is your highest throughput when reading in the QVD optimized files?