Skip to main content
Announcements
Have questions about Qlik Connect? Join us live on April 10th, at 11 AM ET: SIGN UP NOW
cancel
Showing results for 
Search instead for 
Did you mean: 
bgerchikov
Partner - Creator III
Partner - Creator III

Open big document issue

Hello,

We have a big document about 50 GB.

To open this document in "Clear" state or with binary reload  takes about 3-5 minutes. However, if any selection has been made before this document was closed, it might take 1 hour or even more. When the document is coming up, the task manager shows up to 100% CPU utilization for 1-2 minutes and 48GB of memory (out of 384GB), then it fells to 3-4% CPU utilization and waiting.  

CPU is Intel(R) Xeon(R) CPU E5-2690 0 @ 2.90GHz.

Application itself has  8 charts with the average calculation time about 30 seconds.

Would you provide any suggestions?

Thanks!

Boris

14 Replies
bgerchikov
Partner - Creator III
Partner - Creator III
Author

Hello,

We don't use dates at all, there is no section access. 120 out of 180 columns are flags 1/0. App cannot be split, since some calculation use entire data set.  

giakoum
Partner - Master II
Partner - Master II

Ok but still some field is taking a lot of space. i would expect an application of 50GB in memory to be 5GB on disk not 44GB. Document analyzer could give you a clue if it works on a large document like that

DavidFoster1
Specialist
Specialist

Boris Gerchikov wrote:

Hello,

We don't use dates at all, there is no section access. 120 out of 180 columns are flags 1/0. App cannot be split, since some calculation use entire data set. 


That sounds pretty streamlined.


Can you explain more about the 'since some calculation use entire data set'?


Is this a calculation on the opening sheet?


Do you use set analysis or if statements for these flags?


Have you considered using bitwise operators for some of these flags (assuming some of them are flag sets).

bgerchikov
Partner - Creator III
Partner - Creator III
Author

Hi David,

In simple, we calculate performance for the selected client competitors and show its standoff. That's why we need the full data set.

We think about creating "binary" field (101010011....) which would include most of the flags and use bitwise operators... Not sure it will work faster.

Thanks!

DavidFoster1
Specialist
Specialist

I would recommend that you look at a binary option. It has a similar to the junk dimension approach used in Kimball modelling for high numbers of dimension fields. In QlikView's associative model you could drastically reduce the size of the data model by have a few columns with lots of different integer values versus over a hundred columns with bit values in them.