Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.
Hi all,
I am loading an optimized QVD file of 2.3M records and using CrossTable() to get a tall thin version (130 columns to 40). During the CrossTable() part of the process I get CPU usage like the attached image (approx 20% of most cores). How do I get the process to use all available CPU power? Nothing else is going on with the server - it should be dedicated to this load process.
Thanks,
Jason
HI ,
have you tried by changing the Priority of the Process for the QV.exe to High ?
Regards
Yusuf
Hi Yusuf,
Didn't think of that but just tried it and it made no difference.
Jason
Jason
Could you provide the log file from when you reload the qvw ?
Best Regards, Bill
Hi Bill,
Sure - reload log file attached. This is actually from a smaller test (only 330k records in the original QVD file) but demonstrates the same behaviour.
Thanks,
Jason
Jason
How big in rows and Mbytes [or Gbytes] are your QVD files:
[Note: 337,991 * 36 exactly equals 12,181,960]
What is your RAM data transfer rate, could I approximate it as:
What I am angling at here is that I suspect your bottleneck could well be RAM data transfers. I am not going to claim any knowledge of the QlikView internals re Crosstable and RAM data transfers, but would expect it to be heavy.
The Crosstable from your log takes 243 seconds which all things considered is probably in the order of magnitude one could expect.
Could you mull over the above and give your thoughts ?
Best Regards, Bill
Hi Bill - thank you for your continued interest.
We are using HP ProLiant G8 servers with 256GB RAM and 2 x E2670 CPUs, running Win2k8R2-ENT64-v1.8 OS. I wouldn't know where to look for the RAM transfer rate, but I can ask the server guys tomorrow and see if they can help.
If that is it I wonder if it can be tweaked...!
Thanks again,
Jason
EDIT: I didn't answer your question!
FACT_LATEST_Consolidated.qvd = 73,385KB
FACT_LATEST_Consolidated_FACT.qvd = 520,146KB
It could be that the RAM data transfer rate is a bottleneck like Bill suggests, but I suspect a different cause: Limited multi-threading. You have 32 cores (real or hyperthreaded) of which you use 7.
It could be that the load statement for different internal technical reasons only can be split into seven threads, no more. This would also result in the same behaviour.
HIC
Is there anything I can do about that Henric, do you know?
If it is the limit on number of threads that causes this, then there is nothing you can do.
The script execution is sequential, i.e. in principle single threaded. But QlikView splits each record, when read, into several threads. Then all threads must wait for the last one to finish, until the next record can be read. So it is easy to imagine a situation where you have a table that contain seven fields that need more CPU than the others, so that the load will most of the time be limited to seven threads.
HIC