Object Out of Memory problem

[sczetty]

I know that there is a limit to the amount of data I can cram into an object, and I can do the trial and error to get the object built. What I would like to know is the following:

Is there a way to increase the memory setting for a single chart. (I have a need to provide a chart that in effect is used to download large number of rows, ordered by a cyclic dimension selector) I have hundreds of millions of rows, so I run out of memory very easily when a low level of detail is chosen.

Thanks!

Steve

[#2]

Straight from Qlikview Help

Calculation Memory Settings

All calculated objects (e.g. charts or table boxes) require some extra memory for the calculation. Windows will, if it runs out of physical RAM, start swapping memory to disk. This will slow down the computer considerably. In order to prevent this from happening, QlikView may limit the amount of memory given to any calculated sheet object. When more memory is required, QlikView will display the error message "Out of object memory" or "Too many cells". Please note that the cells require memory also after the calculation, as long as the object is displayed on the active sheet.

In most cases it is wisest to leave the setting of limits to QlikView. QlikView will do so taking into account the available RAM and other factors. Provided that the Ignore Calculation Limits setting has been selected in the Objects page of the User Preferences dialog, all memory limits will be handled automatically. Only if this setting is deselected will the Calculation Memory Settings dialog become available under Chart properties and Table Box Properties. With this dialog you may set hard custom limits to specific sheet objects.

You may choose to increase Memory Size Limit (Bytes) or Maximum Number of Cells until the calculation can be performed. It is also possible to leave the object with insufficient memory. In many cases the calculation will be made possible as the data set is narrowed down by selections in the data.

You may access the Calculation Memory Settings dialog for a calculated sheet object at any time via the Memory... button on the General page of the appropriate Properties dialog.

By marking the check box Automatic you turn on automatic limits for the specific sheet object even if the Ignore Calculation Limits setting has been deselected in the Objects page of the User Preferences dialog.

Reset resets all values to the defaults defined on the Objects page in the the User Preferences: Objects property page.

In the Object box you can see the sheet object ID of the current chart or table box.

OK accepts changes and closes the dialog.

Cancel closes the dialog without saving the changes.

Hope this helps ...

[#3]

Which version/release are you using?

ver 8.5.6299 (and probably prior) has a bug

Bug ID:22214 Internal 2 GB limit on symbol table size in file

Also it would probably be a good idea to:

a). Aggregate some of your data in the load script.You may not need that level of granularity in the chart.

b) if possible calculate expressions in the load script instead of in the chart

[johnw]

mtbit wrote: Also it would probably be a good idea to:
a). Aggregate some of your data in the load script.You may not need that level of granularity in the chart.

I would consider this ONLY as a last resort. I have so far always stored my data at the lowest level of granularity that I think anyone might ever want, and let QlikView itself handle the aggregations. That's what QlikView is there for, and you start losing flexibility quickly when you aggregate yourself. But yes, if all else fails, aggregating your data in the load script WILL solve problems like this.

mtbit wrote:b) if possible calculate expressions in the load script instead of in the chart

This I'll agree with, at least when performance is a major concern. Of course, this won't always work directly. For instance, your chart might show one sum divided by another sum. Doing the sums or sums and division ahead of time requires aggregating in the load script, which I consider a no-no. However, the expression inside of the sum could be very complicated. A complicated expression like that might lend itself well to being done in the script instead of in the chart. I think this is likely to save more CPU than RAM, but for complicated expressions, it's definitely worth a try.

[#5]

So for the case where you have tens of thousands of rows or even hundreds of thousands what would you recommend if you wish to create a straight table with many regular and calculated fields?

A filter could be added to reduce the memory, but if I wish to export the whole table to excel I lose this ability with a filter. How do I overcome object out of memory?

[johnw]

brafferty wrote:How do I overcome object out of memory?

Have you already tried the advice in this thread?

[#7]

I have not tried aggregating in the load script because the expressions I am using are mainly sums or differences (there are just many of them). Also, the data I am gathering is from multiple tables.

Would it be better to create this table within the load script and then just bring those fields from the table over?