topic Out of memory when loading multiple QVD files in QlikView

Out of memory when loading multiple QVD files

Wed, 15 Apr 2009 18:39:31 GMT

I'm trying to load multiple (5) QVD files in my script. Each QVD has 15 000 000 to 20 000 000 lines. Adter a while I'll get a out of memory exception. With my approach everything is loaded into the TRANSACTIONS table. Can it be done in any other way?

Directory;
TRANSACTIONS:
LOAD
Date,
Hour,
CUSTOMERID,
CVOID,
LINES,
SESSIONID,
SUPPLIERID,
SYSTEMUSED
FROM TRANSACTIONS_20*.QVD (qvd);

Thanks!

/magnus

Out of memory when loading multiple QVD files

blaise — Wed, 15 Apr 2009 20:13:00 GMT

5 files, each will 15 to 20 millions of records is quite much 🙂

How much RAM do you have?

Out of memory when loading multiple QVD files

Wed, 15 Apr 2009 21:23:05 GMT

Perhaps if the QVDs were loaded in distinct LOADs the problem would resolve itself? Try loading each in a seperate LOAD statement (they will append if the structures are the same of course).

If that works, you might want to create a loop containing a LOAD something like

FOR

each vFile in filelist ...

Regards,

Gordon

Out of memory when loading multiple QVD files

rwunderlich — Thu, 16 Apr 2009 07:04:52 GMT

Are you getting optimized loads? That is, do you see the "optimized" message in the load progress window?

32bit or 64bit?

-Rob

Out of memory when loading multiple QVD files

Thu, 16 Apr 2009 12:17:37 GMT

I'll try that and see what comes out.

Thanks!

/magnus

Out of memory when loading multiple QVD files

Thu, 16 Apr 2009 12:23:41 GMT

For the moment it's a 32-bit Windows 2003 Server with only 4GB memory.

We will shortly have a 64-bit server with 10GB of memory and perhaps the problem disappears for the moment. However, the data we like to dipslay and analyze with QlikView is growing rapidly so I need to find a solution to this anyway.

For sure, we can buy more hardware but it doesn't solve the core issue.

/magnus

Out of memory when loading multiple QVD files

Thu, 16 Apr 2009 15:20:42 GMT

I believe that 32bit W2003 Server will only allow a task up to 2Gb of memory. There is a trick to force it to use more - a switch /3 on loading server I seem to remember will give server up to 3Gb - but this isnt my area really.

Regards,

Gordon