2 Replies Latest reply: Mar 13, 2016 12:24 AM by Hirish V RSS

    which database to use with PE and big datasets

    Thorsten Alxneit

      Dear all,

       

      until now I used Qlikview importing large csv files and to make dashboards, analytics based on that data.

      As, for example, I make a weather analytics of the last 30 years, the csv files are getting quite big and sometimes I yet got the error - running out of memory...

      So I was wondering if it makes sense to set up (e.g.) a sql database?

      Do you have any further documentation on how to work with "stand alone" databases?

       

      Kind regards and thank you

      Thorsten

        • Re: which database to use with PE and big datasets
          Petter Skjolden

          You mean running out of memory while loading the data from the CSV in a load script?

           

          If that is the case - the loading of the data into QlikView in-memory through a SQL-database will not make a difference. Using a SQL-database as a source does not help on memory. I

           

          f you on the other hand use Direct Dicovery for your fact table then using a SQL-database will help. But it comes with some trade-offs and limitations. So I am pretty sure that doing the following two things first will be most useful for you:

           

          1)  Look at how the data you bring in can be made less memory consuming.

          2) Buy more memory for your computer

          • Re: which database to use with PE and big datasets
            Hirish V

            Hi,

             

            • Are you converting you data into QVD and Making use of them in your App.

             

            • Try using incremental load beacause .I hope the old data of weather will not have a change.This may decrease your Reload time and only new records will be appended.

             

            • CSV's -the number of records is limited to 1048576 rows.

             

            • I think in your csv export you can create a export based on monthly or quaterly CSV such that the records will be lesser than the limitation.

             

            HTH,

            Hirish