Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.
Hi,
I am new to Qlik. I am facing an issue while loading a huge CSV file into Qlik. My approach is first to generate a QVD file. But generation of this file is taking too long aprox 20 mins. The data size is 1.2 GB and my system RAM is 8 Gb.
Please provide me some suggestions which can expedite my process
Yes I guess I have the mentioned requirements for running qlikview on desktop
A 1.2GB file will take a long time to process. It also depends on how complex the data is, as QV tries to compress the data as much as possible when it saves to a QVD. Remember that QV will try to load all of the data in memory to speed up processing the QVW.
See also: Symbol Tables and Bit-Stuffed Pointers
So are there any best practices which I can use
I am not loading the file through network it is in my local. Further I am in development mode not on the server
Hi,
1.2 GB in plain text and for only one file is a lot of data, your timing is not so bad since you are not running on a server.
if you have a CSV that big maybe you can split it into smaller QVDs storing latest data in one and the historic in another, etc.
You can also use an incremental load, the first load will take 20 minutes as you said, but after it will only load the new records.
Please let me know if this helps.
Kind regards,
Do not use WHERE filters when LOADing from text files. Do all filtering later on a RESIDENT copy of your table. In that case, you may need more RAM...
I agree with Santiago that the best thing you can do to speed up the loading of large text files is to process the data incrementally.
You can significantly reduce the storage and RAM requirements for QlikView by reducing the number of unique values you are storing.
For example storing dates instead of timestamps where the time component is not required, ofr separating a timestamp value into two separate date and time fields if the time component is required.
This blog has more details. The Importance Of Being Distinct