2 Replies Latest reply: Mar 13, 2016 12:24 AM by Hirish V RSS

    which database to use with PE and big datasets

      Dear all,


      until now I used Qlikview importing large csv files and to make dashboards, analytics based on that data.

      As, for example, I make a weather analytics of the last 30 years, the csv files are getting quite big and sometimes I yet got the error - running out of memory...

      So I was wondering if it makes sense to set up (e.g.) a sql database?

      Do you have any further documentation on how to work with "stand alone" databases?


      Kind regards and thank you


        • Re: which database to use with PE and big datasets
          Petter Skjolden

          You mean running out of memory while loading the data from the CSV in a load script?


          If that is the case - the loading of the data into QlikView in-memory through a SQL-database will not make a difference. Using a SQL-database as a source does not help on memory. I


          f you on the other hand use Direct Dicovery for your fact table then using a SQL-database will help. But it comes with some trade-offs and limitations. So I am pretty sure that doing the following two things first will be most useful for you:


          1)  Look at how the data you bring in can be made less memory consuming.

          2) Buy more memory for your computer

          • Re: which database to use with PE and big datasets
            Hirish V



            • Are you converting you data into QVD and Making use of them in your App.


            • Try using incremental load beacause .I hope the old data of weather will not have a change.This may decrease your Reload time and only new records will be appended.


            • CSV's -the number of records is limited to 1048576 rows.


            • I think in your csv export you can create a export based on monthly or quaterly CSV such that the records will be lesser than the limitation.