RAM explodes - bug?

[#1]

Hi all.

Two tables are to be conjoined by a CONCATENATE.

The first table exists of 6 mio. rows and the second of 6½ mio. rows.

When doing a normal CONCATENATE the RAM explodes (+60 GB), but when doing the same CONCATENATE with a FIRST-command (FIRST 1000000000) on the second table the RAM-usage is as expected (about 5 GB).

We are running QV 11.2 - any known reason for this behavior, besides a bug?

[#2]

Danke,

bin gerade zurück (vom Kaffee ;-)).

Verson 11.2 SR3,

RR

[rbecher]

Cannot reproduce with 11.2 SR5

[rbecher]

Could you try this script and compare the memory usage:

Temp:

LOAD

A,

B,

C

FROM A1.qvd (qvd) ;

LOAD

A,

B,

Null() as C

FROM A2.qvd (qvd) ;

Concatenate should not needed in this case..

- Ralf

[rwunderlich]

Roland,

I think the difference you are seeing is optimized/non-optimized loading of the QVD. There are a few features that are ignored in an optimized load like

SAMPLE

MAP USING

FIRST seems to fit into this category, but only for the second load. Odd.

-Rob

[rwunderlich]

Cap,

I can guess that this is a known behavior. It was my very first blog post back in 2008.

http://qlikviewnotes.blogspot.com/2008/05/when-less-data-means-more-ram.html

My guess is that in the larger case you are getting an un-optimized load and one or more numeric fields is getting unwrapped into strings. You can confirm that by checking field sizes using a mem file + (QV Optimizer) or Document Analyzer.

Can't explain why the FIRST prefix makes it better. Is there a WHERE clause involved in one of the LOADs?

Can you post the entire script?

-Rob

http://masterssummit.com

[#7]

Hi Rob.

Thanks. I can's post the script unfortunately. I read your blog post, but I'm not sure if it explains the case entirely. I think I will have a talk with QV support.

To sum up the behavior, when RAM- and time-usage explodes:

The first table (with lots of fields and 6 mio rows) is loaded un-optimized - loaded alone, this table-load makes no problems (5 GB RAM usage).

The second table (with only 3 fields and 6½ mio rows) is loaded optimized (no where clause or things like that), but with a concatenate in front here RAM explodes and the load takes forever. Loaded alone, this table-load makes no problems (5 GB RAM usage).

1. Putting FIRST on the second load "solves" the RAM- and the time-problem.

2. Putting a where 1 = 1 doesn't help.

3. Changing the script to use the qv auto-concatenate also "solves" the RAM-problem.

Looks to me that there is something rather "un-optimized" about the CONCATENATE-function in some cases.

[#8 · Accepted solution]

It is a bug in QV11 SR2 - upgrading to SR5 could be a solution (work around: use a FIRST-command in the script):

Bug 60817: Memory consumption very high in 11.20 on a machine with high number of cores

According to the more detailed description:

Reloading a document on a machine with less numbers of cores leaves a memory usage of 21.5gb post reload, reloading the same document on a machine with a high number of cores consumes 110 gb of memory.   The application uses a concatenate load.