3 Replies Latest reply: Oct 16, 2010 12:28 PM by Roland Kunle RSS

    Error loading huge amount of data

      Hi,

      I'm trying to load the Apache and IIS logfiles from different servers but Igot an error:

      "Out of Virtual and/or logical memory, allocating 2MB"

       

      I've read in previous posts that it could be because of the data. In that case, all data come from text files that I try to import using Qlikview.

      I've also increase the virtual memory with no results....

       

      There are 4 million entries / month aprox. from each system and I can't load data from 2 systems.

       

      How can I load and get info from this kind of sources?

       

      Thank you in advance!

       

        • Error loading huge amount of data

          Hi,

          The issue you describe is a typical lack of RAM memory issues. You can try to improve your script in terms f memory usage, meaning loading one table a time, storing in qvd and drop the table afterwards. Once you have all the qvds, try to load them in a consolidated model, if this does not work for you means that you will have to buy some more RAM memory for the server.

          Regards.

          • Error loading huge amount of data

            Hi Qlikview-tester,

            the amount of load is independent of your source. Wether RDBMS or textfiles isn't important. QV works fine with very redundant data, in this case the compress-rate may be very high. Do you need every field etc. ?

            Ivan told you some good workarounds but I think you will need sooner or later a machine with a 64bit-Windows and the according RAM. Your data is growing fast ervery month.

             

            Regards

            Roland