topic Re: RAM explodes - bug? in QlikView

RAM explodes - bug?

Sun, 23 Feb 2014 10:15:26 GMT

Hi all.

Two tables are to be conjoined by a CONCATENATE.

The first table exists of 6 mio. rows and the second of 6½ mio. rows.

When doing a normal CONCATENATE the RAM explodes (+60 GB), but when doing the same CONCATENATE with a FIRST-command (FIRST 1000000000) on the second table the RAM-usage is as expected (about 5 GB).

We are running QV 11.2 - any known reason for this behavior, besides a bug?

Re: RAM explodes - bug?

nilesh_gangurde — Sun, 23 Feb 2014 12:10:53 GMT

its not a bug.

your script is going wrong somewhere.

you can check the script that u have all the fields with same name in the both table.

also check your datamodel, that after concatenation there should be only one table.

-Nilesh

Re: RAM explodes - bug?

Sun, 23 Feb 2014 12:32:33 GMT

Hi Nilesh.

Thanks, but names are uniform, and only one table is left after the concatenate.

I'm not sure how you know it's not a bug. It's working with an arbitrary FIRST-command added, that is a little strange to me. Without the FIRST-command the RAM-usage explodes, and not because the script is loading different or more data.

Temp:

LOAD

FROM A1.qvd (qvd) ;

CONCATENATE (Temp)

FIRST 1000000

LOAD

FROM A2.qvd (qvd) ;

Re: RAM explodes - bug?

Sun, 23 Feb 2014 13:07:08 GMT

Hi,

a little note:

in your post you wrote First 100 Mio

in your script you have First 1 (one) Mio

Only typo? If not, everthing is as expected.

Re: RAM explodes - bug?

Sun, 23 Feb 2014 13:22:19 GMT

Hi Roland.

Thanks. Just a typo/example script. In the live script it's: FIRST 100000000

Re: RAM explodes - bug?

rbecher — Sun, 23 Feb 2014 13:48:45 GMT

..are you sure with the amount of rows in the 2nd QVD?

Re: RAM explodes - bug?

Sun, 23 Feb 2014 13:58:12 GMT

ok, this will not resolve your problem, but I think it is worth to be posted:

Meanwhile i was a bit testing and discoverd some funny things:

When concatenating a second qvd to a just loaded first one, the First-Statement is ignored. "First 1" should add the very first record but QV added all rows from the second qvd. "First 1000" same.

Then I tried different things and considered that after adding a Where - Clause the First-Statement worked fine. Even if the Where was a "Where 1=1".

Loading only one qvd, "First" works also fine.

HtH

Roland

P.s: using 11.20 SR3

Re: RAM explodes - bug?

Sun, 23 Feb 2014 13:59:32 GMT

Hi Ralf,

es sind doch immer wieder die selben, gelle !

Gruß

Roland

P.S:

Jetzt aber ab zum Kaffee.

Re: RAM explodes - bug?

rbecher — Sun, 23 Feb 2014 18:11:13 GMT

Interesting. Another phenomena.. Which QlikView version is affected?

Re: RAM explodes - bug?

rbecher — Sun, 23 Feb 2014 18:13:43 GMT

Grüße zurück..

Re: RAM explodes - bug?

Sun, 23 Feb 2014 18:25:00 GMT

Danke,

bin gerade zurück (vom Kaffee ;-)).

Verson 11.2 SR3,

Re: RAM explodes - bug?

rbecher — Sun, 23 Feb 2014 18:43:46 GMT

Cannot reproduce with 11.2 SR5

Re: RAM explodes - bug?

rbecher — Sun, 23 Feb 2014 18:45:20 GMT

Could you try this script and compare the memory usage:

Temp:

LOAD

FROM A1.qvd (qvd) ;

LOAD

Null() as C

FROM A2.qvd (qvd) ;

Concatenate should not needed in this case..

- Ralf

Re: RAM explodes - bug?

rwunderlich — Sun, 23 Feb 2014 22:34:50 GMT

Roland,

I think the difference you are seeing is optimized/non-optimized loading of the QVD. There are a few features that are ignored in an optimized load like

SAMPLE

MAP USING

FIRST seems to fit into this category, but only for the second load. Odd.

-Rob

Re: RAM explodes - bug?

rwunderlich — Sun, 23 Feb 2014 22:44:57 GMT

Cap,

I can guess that this is a known behavior. It was my very first blog post back in 2008.

http://qlikviewnotes.blogspot.com/2008/05/when-less-data-means-more-ram.html

My guess is that in the larger case you are getting an un-optimized load and one or more numeric fields is getting unwrapped into strings. You can confirm that by checking field sizes using a mem file + (QV Optimizer) or Document Analyzer.

Can't explain why the FIRST prefix makes it better. Is there a WHERE clause involved in one of the LOADs?

Can you post the entire script?

-Rob

http://masterssummit.com

Re: RAM explodes - bug?

Mon, 24 Feb 2014 08:06:00 GMT

Hi Rob.

Thanks. I can's post the script unfortunately. I read your blog post, but I'm not sure if it explains the case entirely. I think I will have a talk with QV support.

To sum up the behavior, when RAM- and time-usage explodes:

The first table (with lots of fields and 6 mio rows) is loaded un-optimized - loaded alone, this table-load makes no problems (5 GB RAM usage).

The second table (with only 3 fields and 6½ mio rows) is loaded optimized (no where clause or things like that), but with a concatenate in front here RAM explodes and the load takes forever. Loaded alone, this table-load makes no problems (5 GB RAM usage).

1. Putting FIRST on the second load "solves" the RAM- and the time-problem.

2. Putting a where 1 = 1 doesn't help.

3. Changing the script to use the qv auto-concatenate also "solves" the RAM-problem.

Looks to me that there is something rather "un-optimized" about the CONCATENATE-function in some cases.

Re: RAM explodes - bug?

Fri, 28 Feb 2014 10:40:07 GMT

It is a bug in QV11 SR2 - upgrading to SR5 could be a solution (work around: use a FIRST-command in the script):

Bug 60817: Memory consumption very high in 11.20 on a machine with high number of cores

According to the more detailed description:

Reloading a document on a machine with less numbers of cores leaves a memory usage of 21.5gb post reload, reloading the same document on a machine with a high number of cores consumes 110 gb of memory. The application uses a concatenate load.