Skip to main content
Announcements
Have questions about Qlik Connect? Join us live on April 10th, at 11 AM ET: SIGN UP NOW
cancel
Showing results for 
Search instead for 
Did you mean: 
Anonymous
Not applicable

Data Upload taking too long

Hi,

I am new to Qlik. I am facing an issue while loading a huge CSV file into Qlik. My approach is first to generate a QVD file. But generation of this file is taking too long aprox 20 mins. The data size is 1.2 GB and my system RAM is 8 Gb.

Please provide me some suggestions which can expedite my process

14 Replies
Anonymous
Not applicable
Author

Yes I guess I have the mentioned requirements for running qlikview on desktop

ThornOfCrowns
Specialist II
Specialist II

A 1.2GB file will take a long time to process. It also depends on how complex the data is, as QV tries to compress the data as much as possible when it saves to a QVD. Remember that QV will try to load all of the data in memory to speed up processing the QVW.

See also: Symbol Tables and Bit-Stuffed Pointers

Clever_Anjos
Employee
Employee

  • 8Gb is not enough for a real Qlik deployment
  • Are you loading that file through network or local file?
  • Are there more processes running at same time?
  • Which version are you Qlik and Windows Server?
Anonymous
Not applicable
Author

So are there any best practices which I can use

Anonymous
Not applicable
Author

I am not loading the file through network it is in my local. Further I am in development mode not on the server

santiago_respane
Specialist
Specialist

Hi,

1.2 GB in plain text and for only one file is a lot of data, your timing is not so bad since you are not running on a server.

if you have a CSV that big maybe you can split it into smaller QVDs storing latest data in one and the historic in another, etc.

You can also use an incremental load, the first load will take 20 minutes as you said, but after it will only load the new records.

Please let me know if this helps.

Kind regards,

Peter_Cammaert
Partner - Champion III
Partner - Champion III

Do not use WHERE filters when LOADing from text files. Do all filtering later on a RESIDENT copy of your table. In that case, you may need more RAM...

I agree with Santiago that the best thing you can do to speed up the loading of large text files is to process the data incrementally.

Colin-Albert

You can significantly reduce the storage and RAM requirements for QlikView by reducing the number of unique values you are storing.

For example storing dates instead of timestamps where the time component is not required,  ofr separating a timestamp value into two separate date and time fields if the time component is required.

This blog has more details.  The Importance Of Being Distinct