How can we get QV.exe to use more CPU?

[Jason_Michaelides]

Hi all,

I am loading an optimized QVD file of 2.3M records and using CrossTable() to get a tall thin version (130 columns to 40).  During the CrossTable() part of the process I get CPU usage like the attached image (approx 20% of most cores). How do I get the process to use all available CPU power?  Nothing else is going on with the server - it should be dedicated to this load process.

Thanks,

Jason

[#2]

HI ,

have you tried by changing the Priority of the Process for the QV.exe to High ?

Regards

Yusuf

[Jason_Michaelides]

Hi Yusuf,

Didn't think of that but just tried it and it made no difference.

Jason

[Anonymous]

Jason

Could you provide the log file from when you reload the qvw ?

Best Regards,     Bill

[Jason_Michaelides]

Hi Bill,

Sure - reload log file attached. This is actually from a smaller test (only 330k records in the original QVD file) but demonstrates the same behaviour.

Thanks,

Jason

[Anonymous]

Jason

How big in rows and Mbytes [or Gbytes] are your QVD files:

• D:\DEV FILES\JASON MICHAELIDES\QlikView\DEV QVDs\FACT_LATEST_Consolidated.qvd
  • 337,991 rows, I suspect from your log file, but how many Mbytes ?

• D:\DEV FILES\JASON MICHAELIDES\QlikView\DEV QVDs\FACT_LATEST_Consolidated_FACT.qvd
  • 12,181,960 rows, I suspect from your log file, but how many Mbytes ?

[Note:     337,991 * 36 exactly equals 12,181,960]

What is your RAM data transfer rate, could I approximate it as:

• 1 billion data transfers / second or 10 Gbytes /sec
  • [Please excuse this very rough guess approximation]

What I am angling at here is that I suspect your bottleneck could well be RAM data transfers.  I am not going to claim any knowledge of the QlikView internals re Crosstable and RAM data transfers, but would expect it to be heavy.

The Crosstable from your log takes 243 seconds which all things considered is probably in the order of magnitude one could expect.

Could you mull over the above and give your thoughts ?

Best Regards,     Bill

[Jason_Michaelides]

Hi Bill - thank you for your continued interest.

We are using HP ProLiant G8 servers with 256GB RAM and 2 x E2670 CPUs, running Win2k8R2-ENT64-v1.8 OS. I wouldn't know where to look for the RAM transfer rate, but I can ask the server guys tomorrow and see if they can help.

If that is it I wonder if it can be tweaked...!

Thanks again,

Jason

EDIT: I didn't answer your question!

FACT_LATEST_Consolidated.qvd = 73,385KB

FACT_LATEST_Consolidated_FACT.qvd = 520,146KB

[hic]

It could be that the RAM data transfer rate is a bottleneck like Bill suggests, but I suspect a different cause: Limited multi-threading. You have 32 cores (real or hyperthreaded) of which you use 7.

It could be that the load statement for different internal technical reasons only can be split into seven threads, no more. This would also result in the same behaviour.

HIC

[Jason_Michaelides]

Is there anything I can do about that Henric, do you know?

[hic]

If it is the limit on number of threads that causes this, then there is nothing you can do.

The script execution is sequential, i.e. in principle single threaded. But QlikView splits each record, when read, into several threads. Then all threads must wait for the last one to finish, until the next record can be read. So it is easy to imagine a situation where you have a table that contain seven fields that need more CPU than the others, so that the load will most of the time be limited to seven threads.

HIC