topic Re: Ram consumtion on script calculation in QlikView

Ram consumtion on script calculation

Anonymous — Thu, 07 Nov 2013 15:57:40 GMT

Hello

unfortunately I have not enough RAM to make a calculation. All optimizations have been done.

The script looks like this

Load

Key1*100000+Key2 as Masterkey,

Key1,

Key2,

SomeotherData

From some.qvd (qvd);

Does it make sence to load in slices and then to merge the qvds?

Something like:

L1:

Load

Key1*100000+Key2 as Masterkey,

Key1,

Key2,

SomeotherData

From some.qvd (qvd) where Key1 < 40000000;

Save L1 to L1.qvd;

Drop Table L1;

L2:

Load

Key1*100000+Key2 as Masterkey,

Key1,

Key2,

SomeotherData

From some.qvd (qvd) where Key1 >= 40000000;

Save L2 to L2.qvd;

Drop Table L2;

Final:

Load * From L1.qvd;

Load * From L2.qvd;

------------

cheers chesterluck

Re: Ram consumtion on script calculation

Clever_Anjos — Thu, 07 Nov 2013 16:01:14 GMT

Unless you´re not using a group by (very CPU and memory intensive) that change won´t change RAM consumption.

You should check your data and maybe do some optimizations:

1) Break a datetime field into a date field and time field

2) Round some decimal values

this will give QlikView more chance to optimize your data.

At last resource, increase RAM

Re: Ram consumtion on script calculation

Clever_Anjos — Thu, 07 Nov 2013 16:02:31 GMT

Another issue could be synthetic keys, do you have any of them inside your app?

Re: Ram consumtion on script calculation

Anonymous — Thu, 07 Nov 2013 16:15:53 GMT

No,

I had before a String as a composite key:

Key1&_&Key2 as Masterkey

but then changed it to Key1*100000+Key2 as Masterkey

Couldnt use AutoNumber() as Masterkey - as I need the MasterKey for an incr. load.

I do not have any more ideas

Load

Key1*100000+Key2 as Masterkey, >num

Key1, >num

Key2, >num

SomeotherData >num

From some.qvd (qvd);

Re: Ram consumtion on script calculation

swuehl — Thu, 07 Nov 2013 16:23:10 GMT

Can you tell us a bit more about your setting (QV version, OS version, amount of RAM, size of qvd)?

Do I read your last post correctly,

Key1&_&Key2 as Masterkey

worked fine, while this

Key1*100000+Key2 as Masterkey,

stops because of running out of memory?

Re: Ram consumtion on script calculation

Anonymous — Thu, 07 Nov 2013 16:30:35 GMT

Chester

Could also provide the log file from a load the fails ?

Best Regards, Bill

Re: Ram consumtion on script calculation

Anonymous — Thu, 07 Nov 2013 16:43:39 GMT

No Key1&_&Key2 as Masterkey didn't work

Optimization to Key1*100000+Key2 as Masterkey, also didn't work.

20GB Ram Win2008Server size of qvd between 2 and 6 GB

Re: Ram consumtion on script calculation

Anonymous — Tue, 12 Nov 2013 12:46:27 GMT

i finally hat to change the script completely. The Masterkey I needed for the incr load.

I generated this key every load anew using Autonumber(). Im testing now the appl. and hope everything works just fine.

Thank you all

chesterluck