until now I used Qlikview importing large csv files and to make dashboards, analytics based on that data.
As, for example, I make a weather analytics of the last 30 years, the csv files are getting quite big and sometimes I yet got the error - running out of memory...
So I was wondering if it makes sense to set up (e.g.) a sql database?
Do you have any further documentation on how to work with "stand alone" databases?
You mean running out of memory while loading the data from the CSV in a load script?
If that is the case - the loading of the data into QlikView in-memory through a SQL-database will not make a difference. Using a SQL-database as a source does not help on memory. I
f you on the other hand use Direct Dicovery for your fact table then using a SQL-database will help. But it comes with some trade-offs and limitations. So I am pretty sure that doing the following two things first will be most useful for you:
1) Look at how the data you bring in can be made less memory consuming.
Are you converting you data into QVD and Making use of them in your App.
Try using incremental load beacause .I hope the old data of weather will not have a change.This may decrease your Reload time and only new records will be appended.
CSV's -the number of records is limited to 1048576 rows.
I think in your csv export you can create a export based on monthly or quaterly CSV such that the records will be lesser than the limitation.
