Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.
Hello,
We have a big document about 50 GB.
To open this document in "Clear" state or with binary reload takes about 3-5 minutes. However, if any selection has been made before this document was closed, it might take 1 hour or even more. When the document is coming up, the task manager shows up to 100% CPU utilization for 1-2 minutes and 48GB of memory (out of 384GB), then it fells to 3-4% CPU utilization and waiting.
CPU is Intel(R) Xeon(R) CPU E5-2690 0 @ 2.90GHz.
Application itself has 8 charts with the average calculation time about 30 seconds.
Would you provide any suggestions?
Thanks!
Boris
Hello,
We don't use dates at all, there is no section access. 120 out of 180 columns are flags 1/0. App cannot be split, since some calculation use entire data set.
Ok but still some field is taking a lot of space. i would expect an application of 50GB in memory to be 5GB on disk not 44GB. Document analyzer could give you a clue if it works on a large document like that
Boris Gerchikov wrote:
Hello,
We don't use dates at all, there is no section access. 120 out of 180 columns are flags 1/0. App cannot be split, since some calculation use entire data set.
That sounds pretty streamlined.
Can you explain more about the 'since some calculation use entire data set'?
Is this a calculation on the opening sheet?
Do you use set analysis or if statements for these flags?
Have you considered using bitwise operators for some of these flags (assuming some of them are flag sets).
Hi David,
In simple, we calculate performance for the selected client competitors and show its standoff. That's why we need the full data set.
We think about creating "binary" field (101010011....) which would include most of the flags and use bitwise operators... Not sure it will work faster.
Thanks!
I would recommend that you look at a binary option. It has a similar to the junk dimension approach used in Kimball modelling for high numbers of dimension fields. In QlikView's associative model you could drastically reduce the size of the data model by have a few columns with lots of different integer values versus over a hundred columns with bit values in them.