topic Re: Open big document issue in QlikView

Open big document issue

bgerchikov — Mon, 26 Jan 2026 18:19:17 GMT

Hello,

We have a big document about 50 GB.

To open this document in "Clear" state or with binary reload takes about 3-5 minutes. However, if any selection has been made before this document was closed, it might take 1 hour or even more. When the document is coming up, the task manager shows up to 100% CPU utilization for 1-2 minutes and 48GB of memory (out of 384GB), then it fells to 3-4% CPU utilization and waiting.

CPU is Intel(R) Xeon(R) CPU E5-2690 0 @ 2.90GHz.

Application itself has 8 charts with the average calculation time about 30 seconds.

Would you provide any suggestions?

Thanks!

Boris

Re: Open big document issue

mrybalko — Tue, 05 Aug 2014 14:30:32 GMT

Hello, Boris

I suggest to split this document using document chaining.

More information Best Practices for Data Modelling (page 19)

Re: Open big document issue

giakoum — Tue, 05 Aug 2014 14:30:51 GMT

This is a really big app. What is the app size on disk?

Re: Open big document issue

DavidFoster1 — Tue, 05 Aug 2014 14:35:13 GMT

That is an impressive document .

To improve first page rendering try:

Avoid tables that show more than 1000 rows without a calculation condition.
Have a filter condition (e.g. current year) triggered OnOpen.
Run the document through Rob Wunderlich Document Analyser to see if you can remove any unneeded data.

You didn't mention how many CPUs and Cores your server has. Given a very rough rule-of-thumb of 8GB per core you should have 48 cores (so 8 6-core CPUs or something like that)

Re: Open big document issue

bgerchikov — Tue, 05 Aug 2014 14:45:06 GMT

Thanks!

Here are some answers:

Document has been analyzed, redundant fields have been removed... all data in one fact table (550 million records, 180 columns) + 1 small (60 records) dimension. Size on the disk is 44GB

The server has 16 cores - you mean it's not enough?. Filter on the open doesn't help.

Re: Open big document issue

giakoum — Tue, 05 Aug 2014 14:47:15 GMT

Both rows and columns are a lot. 180 columns and you use them all? What do they contain? text? how many distinct values per column? if you cannot reduce columns and distinct values, you should seriously consider splitting the application.

Re: Open big document issue

marcus_sommer — Tue, 05 Aug 2014 14:47:57 GMT

If there are high-cardinality fields in your app like row-id's or timestamps - you could reduce the app-size (and open-time) a lot by splitting these fields into two or more fields, like these patterns:

date(floor(Timestamp), FORMAT) as Date,

time(frac(Timestamp), FORMAT) as Time

This logic could be applied by other numeric or string fields, too.

- Marcus

Re: Open big document issue

giakoum — Tue, 05 Aug 2014 14:48:45 GMT

By the way, on disk it is 44 and in memory near 50GB? What compression format are you using?

Re: Open big document issue

jfkinspari — Tue, 05 Aug 2014 14:48:48 GMT

Do you use Section Access in the document?

This can increase load time while binary loads are unaffected.

Re: Open big document issue

DavidFoster1 — Tue, 05 Aug 2014 14:52:39 GMT

180 columns! Ouch!

When you are dealing with those kinds of row volumes then you need to be looking at 20-30 columns no more. Especially if you have lots of text values.

Do you have a table/chart that tries to show more than 1000 rows/bars/lines/dots on open?

Is this a VM or physical server?

Re: Open big document issue

bgerchikov — Tue, 05 Aug 2014 14:54:05 GMT

Hello,

We don't use dates at all, there is no section access. 120 out of 180 columns are flags 1/0. App cannot be split, since some calculation use entire data set.

Re: Open big document issue

giakoum — Tue, 05 Aug 2014 14:58:42 GMT

Ok but still some field is taking a lot of space. i would expect an application of 50GB in memory to be 5GB on disk not 44GB. Document analyzer could give you a clue if it works on a large document like that

Re: Open big document issue

DavidFoster1 — Tue, 05 Aug 2014 14:59:31 GMT

Boris Gerchikov wrote:

Hello,

We don't use dates at all, there is no section access. 120 out of 180 columns are flags 1/0. App cannot be split, since some calculation use entire data set.

That sounds pretty streamlined.

Can you explain more about the 'since some calculation use entire data set'?

Is this a calculation on the opening sheet?

Do you use set analysis or if statements for these flags?

Have you considered using bitwise operators for some of these flags (assuming some of them are flag sets).

Re: Open big document issue

bgerchikov — Tue, 05 Aug 2014 15:10:25 GMT

Hi David,

In simple, we calculate performance for the selected client competitors and show its standoff. That's why we need the full data set.

We think about creating "binary" field (101010011....) which would include most of the flags and use bitwise operators... Not sure it will work faster.

Thanks!

Re: Open big document issue

DavidFoster1 — Tue, 05 Aug 2014 15:16:40 GMT

I would recommend that you look at a binary option. It has a similar to the junk dimension approach used in Kimball modelling for high numbers of dimension fields. In QlikView's associative model you could drastically reduce the size of the data model by have a few columns with lots of different integer values versus over a hundred columns with bit values in them.