Qlik Community

QlikView Scripting

Discussion Board for collaboration on QlikView Scripting.

Announcements
QlikView Fans! We’d love to hear from you.
Share your QlikView feedback with the product team… Click here to participate in our 5-minute survey.
Rules, plus terms and conditions, can be found here.
Not applicable

which database to use with PE and big datasets

Dear all,

until now I used Qlikview importing large csv files and to make dashboards, analytics based on that data.

As, for example, I make a weather analytics of the last 30 years, the csv files are getting quite big and sometimes I yet got the error - running out of memory...

So I was wondering if it makes sense to set up (e.g.) a sql database?

Do you have any further documentation on how to work with "stand alone" databases?

Kind regards and thank you

Thorsten

2 Replies
MVP
MVP

Re: which database to use with PE and big datasets

You mean running out of memory while loading the data from the CSV in a load script?

If that is the case - the loading of the data into QlikView in-memory through a SQL-database will not make a difference. Using a SQL-database as a source does not help on memory. I

f you on the other hand use Direct Dicovery for your fact table then using a SQL-database will help. But it comes with some trade-offs and limitations. So I am pretty sure that doing the following two things first will be most useful for you:

1)  Look at how the data you bring in can be made less memory consuming.

2) Buy more memory for your computer

hirishv7
Honored Contributor

Re: which database to use with PE and big datasets

Hi,

  • Are you converting you data into QVD and Making use of them in your App.

  • Try using incremental load beacause .I hope the old data of weather will not have a change.This may decrease your Reload time and only new records will be appended.

  • CSV's -the number of records is limited to 1048576 rows.

  • I think in your csv export you can create a export based on monthly or quaterly CSV such that the records will be lesser than the limitation.

HTH,

Hirish

Community Browser