topic Error loading huge amount of data in QlikView

Error loading huge amount of data

Thu, 14 Oct 2010 16:56:11 GMT

Hi,

I'm trying to load the Apache and IIS logfiles from different servers but Igot an error:

"Out of Virtual and/or logical memory, allocating 2MB"

I've read in previous posts that it could be because of the data. In that case, all data come from text files that I try to import using Qlikview.

I've also increase the virtual memory with no results....

There are 4 million entries / month aprox. from each system and I can't load data from 2 systems.

How can I load and get info from this kind of sources?

Thank you in advance!

Error loading huge amount of data

Thu, 14 Oct 2010 17:01:48 GMT

Hi,

The issue you describe is a typical lack of RAM memory issues. You can try to improve your script in terms f memory usage, meaning loading one table a time, storing in qvd and drop the table afterwards. Once you have all the qvds, try to load them in a consolidated model, if this does not work for you means that you will have to buy some more RAM memory for the server.

Regards.

Error loading huge amount of data

Fri, 15 Oct 2010 19:40:15 GMT

I'm using a 32 bits system with 4GB RAM, so It has the max memory possible...

Is it better to load data from a database than using a text file?

Thanks

Error loading huge amount of data

Sat, 16 Oct 2010 16:28:10 GMT

Hi Qlikview-tester,

the amount of load is independent of your source. Wether RDBMS or textfiles isn't important. QV works fine with very redundant data, in this case the compress-rate may be very high. Do you need every field etc. ?

Ivan told you some good workarounds but I think you will need sooner or later a machine with a 64bit-Windows and the according RAM. Your data is growing fast ervery month.

Regards

Roland