Qlik Community

QlikView App Dev

Discussion Board for collaboration related to QlikView App Development.

Announcements
See why Qlik was recognized for the seventh year in a row – and discover how we can help you tackle your data integration challenges. Get the report
cancel
Showing results for 
Search instead for 
Did you mean: 
sunil2288
Creator III
Creator III

Out of Physical and/or logical memory

HI all

I am using qlikview v9,I am incurring an error while handling 2.3millions of data..The error is showing as Out of Physical and/or logical memory.How can i resolve it. The table having four fields.

Comments awaited..

Thanks and Regards,

Sunil

1 Solution

Accepted Solutions
pover
Luminary Alumni
Luminary Alumni

Hi Sunil,

Is your machine 64 bits or 32 bits? How much memory does your machine have? What data types are you trying to import?

What's your data model look like when you run the script through the debug? Does it have any synthetic tables or are you doing any join between 2 tables that don't share a common field? Any complex group by or interval match statements?

I remember one case in which I successfully loaded about 1.5 million total lines from 5 tables with several columns on a 32 bit machine with 2 Gb of memory RAM. The data model was simple.

Regards.

View solution in original post

30 Replies
pover
Luminary Alumni
Luminary Alumni

Hi Sunil,

Is your machine 64 bits or 32 bits? How much memory does your machine have? What data types are you trying to import?

What's your data model look like when you run the script through the debug? Does it have any synthetic tables or are you doing any join between 2 tables that don't share a common field? Any complex group by or interval match statements?

I remember one case in which I successfully loaded about 1.5 million total lines from 5 tables with several columns on a 32 bit machine with 2 Gb of memory RAM. The data model was simple.

Regards.

sunil2288
Creator III
Creator III
Author

HI karl

My machine is 32 bit and my table is a simple model having 5 fields,One is a date field,and 4 are numerical values.i am having 4gb ram and 2.52 ghz of processor..

Regards

Not applicable

Hello

How much do you allocate for virtual memory. I've changed both my initial and maximum to about 1.5 times my physical memory and I barely ever get any memory problems. I'm on a 32-bit machine with 4 gb (only 3gb active with XP) in memory.

Regards, Max

sunil2288
Creator III
Creator III
Author

Hi max

My virtual memory is having min value 2046 and max value 10000 for c:\ drive. So how to resolve it.

Thanks

pover
Luminary Alumni
Luminary Alumni

The moment QlikView starts using virutal memory, you're better off stopping the reload or graph generation because it is going to take too long. Max is right that XP allows for 3 GB to be used, but any one application can use a maximum of 2 GB. Open the task manager and see how much memory QV is using at the time it sends you the error. It should be around 2 GB.

Does the error show up when you do a reload or when you are generating a graph?

Not applicable

If you reach a maximum of your memory - you are doing something wrong. Review your data structure and expressions.

If you don't reach a maximum of your memory, but still receive 'Out of memory' error, maybe you have a pretty heavy calculation in formulas. In this case you can play with 'Calculated Formulas Max Memory':

1. go to Settings->User Preferences->Object

2. uncheck "Ignore Calculation Limits"

3. set appropriate amount of memory for "Calculated Formulas Max Memory", by default it's 1 MB.

wadesims
Contributor III
Contributor III

You might find it useful to to look at your table and determine the number of unique values in each of the fields. Perhaps the easiest way to identify the number of distinct records in a table is to go to [Settings] [Document Properties] [Tables] and look at (or export) the structure of the tables. The important columns to look at are the # Values and # Distinct. Because the amount of memory required to store data is proportional to the number of distinct values, and the field with the largest number of distinct values is where you want to look first.

You indicated that you have a date field. If you are storing the values as a datetime stamp and only want the date, you might consider truncating the time portion from that field. If you need both the date and the time, you could split them into separate columns. For example, if you have 2.5 million rows where each has a distinct datetime stamp it can eat up a lot of memory. In our case, we only wanted the date and the hour associated with a transaction so we converted the datetime as follows (SQL Server syntax):

CAST(CAST(TranDTime AS BIGINT) AS SMALLDATETIME) AS TranDate,
DATEPART(hour, TranDTime) AS TranHour,

By splitting the datetime into two fields we were able to reduce the memory required to store that field.

I hope that this helps!

- Wade

sunil2288
Creator III
Creator III
Author

yes at the time reload also its generating the error..

pover
Luminary Alumni
Luminary Alumni

Hi Sunil,

Load just one column and see if you get the out of memory error. If not then load two columns, etc. until you get the error. That might tell us if one column in particular is giving you problems.

Anyway you can post your script?

Regards.