Qlik Community

QlikView Deployment

Discussion Board for collaboration related to QlikView Deployment.

khan_fayez
New Contributor III

Data Upload taking too long

Hi,

I am new to Qlik. I am facing an issue while loading a huge CSV file into Qlik. My approach is first to generate a QVD file. But generation of this file is taking too long aprox 20 mins. The data size is 1.2 GB and my system RAM is 8 Gb.

Please provide me some suggestions which can expedite my process

14 Replies
sureshqv
Esteemed Contributor III

Re: Data Upload taking too long

khan_fayez
New Contributor III

Re: Data Upload taking too long

Yes I guess I have the mentioned requirements for running qlikview on desktop

thornofcrowns
Valued Contributor II

Re: Data Upload taking too long

A 1.2GB file will take a long time to process. It also depends on how complex the data is, as QV tries to compress the data as much as possible when it saves to a QVD. Remember that QV will try to load all of the data in memory to speed up processing the QVW.

See also: Symbol Tables and Bit-Stuffed Pointers

Employee
Employee

Re: Data Upload taking too long

  • 8Gb is not enough for a real Qlik deployment
  • Are you loading that file through network or local file?
  • Are there more processes running at same time?
  • Which version are you Qlik and Windows Server?
khan_fayez
New Contributor III

Re: Data Upload taking too long

So are there any best practices which I can use

khan_fayez
New Contributor III

Re: Data Upload taking too long

I am not loading the file through network it is in my local. Further I am in development mode not on the server

santiago_respane
Valued Contributor

Re: Data Upload taking too long

Hi,

1.2 GB in plain text and for only one file is a lot of data, your timing is not so bad since you are not running on a server.

if you have a CSV that big maybe you can split it into smaller QVDs storing latest data in one and the historic in another, etc.

You can also use an incremental load, the first load will take 20 minutes as you said, but after it will only load the new records.

Please let me know if this helps.

Kind regards,

Re: Data Upload taking too long

Do not use WHERE filters when LOADing from text files. Do all filtering later on a RESIDENT copy of your table. In that case, you may need more RAM...

I agree with Santiago that the best thing you can do to speed up the loading of large text files is to process the data incrementally.

Re: Data Upload taking too long

You can significantly reduce the storage and RAM requirements for QlikView by reducing the number of unique values you are storing.

For example storing dates instead of timestamps where the time component is not required,  ofr separating a timestamp value into two separate date and time fields if the time component is required.

This blog has more details.  The Importance Of Being Distinct

Community Browser