topic Re: Data Upload taking too long in QlikView

Data Upload taking too long

Anonymous — Mon, 26 Jan 2026 18:19:17 GMT

Hi,

I am new to Qlik. I am facing an issue while loading a huge CSV file into Qlik. My approach is first to generate a QVD file. But generation of this file is taking too long aprox 20 mins. The data size is 1.2 GB and my system RAM is 8 Gb.

Please provide me some suggestions which can expedite my process

Re: Data Upload taking too long

Chanty4u — Wed, 25 May 2016 11:30:18 GMT

hi,

System Requirements for QlikView ‒ QlikView

Re: Data Upload taking too long

Anonymous — Wed, 25 May 2016 11:37:21 GMT

Yes I guess I have the mentioned requirements for running qlikview on desktop

Re: Data Upload taking too long

ThornOfCrowns — Wed, 25 May 2016 11:42:33 GMT

A 1.2GB file will take a long time to process. It also depends on how complex the data is, as QV tries to compress the data as much as possible when it saves to a QVD. Remember that QV will try to load all of the data in memory to speed up processing the QVW.

Re: Data Upload taking too long

Clever_Anjos — Wed, 25 May 2016 11:42:45 GMT

8Gb is not enough for a real Qlik deployment
Are you loading that file through network or local file?
Are there more processes running at same time?
Which version are you Qlik and Windows Server?

Re: Data Upload taking too long

Anonymous — Wed, 25 May 2016 11:48:13 GMT

So are there any best practices which I can use

Re: Data Upload taking too long

Anonymous — Wed, 25 May 2016 11:49:37 GMT

I am not loading the file through network it is in my local. Further I am in development mode not on the server

Re: Data Upload taking too long

santiago_respane — Wed, 25 May 2016 11:51:20 GMT

Hi,

1.2 GB in plain text and for only one file is a lot of data, your timing is not so bad since you are not running on a server.

if you have a CSV that big maybe you can split it into smaller QVDs storing latest data in one and the historic in another, etc.

You can also use an incremental load, the first load will take 20 minutes as you said, but after it will only load the new records.

Please let me know if this helps.

Kind regards,

Re: Data Upload taking too long

Peter_Cammaert — Wed, 25 May 2016 12:22:14 GMT

Do not use WHERE filters when LOADing from text files. Do all filtering later on a RESIDENT copy of your table. In that case, you may need more RAM...

I agree with Santiago that the best thing you can do to speed up the loading of large text files is to process the data incrementally.

Re: Data Upload taking too long

Colin-Albert — Wed, 25 May 2016 12:29:57 GMT

You can significantly reduce the storage and RAM requirements for QlikView by reducing the number of unique values you are storing.

For example storing dates instead of timestamps where the time component is not required, ofr separating a timestamp value into two separate date and time fields if the time component is required.

This blog has more details. The Importance Of Being Distinct

Re: Data Upload taking too long

vinafidalgo — Wed, 25 May 2016 13:23:14 GMT

The time against the file size seems normal to me.

Re: Data Upload taking too long

Peter_Cammaert — Wed, 25 May 2016 15:23:39 GMT

Well, if text files would have been lightning-fast all along, nobody would have had the crazy idea to invent all sorts of databases. I guess that keeping your data simple & readable comes with a performance disadvantage. A huge one.

If you cannot change the packaging of your data or the nature of the transfer, the best solution is maybe to buy a system with an extremely fast I/O-channel. SSD's aren't that exclusive or expensive anymore. A server has better I/O performance by offering faster interfaces and and in many cases parallelizing disk access...

Peter

Re: Data Upload taking too long

Colin-Albert — Wed, 25 May 2016 15:30:24 GMT

Have you tried using Debug mode with a limited load of say 1000 rows to test that your data load is creating sensible data?

or add a First 1000 command before your load to test the data.

First 1000

load

......

from ...

Re: Data Upload taking too long

Anonymous — Thu, 26 May 2016 07:08:32 GMT

‌yes I did it First 1000.My thinking was can I implement some kind of best practices like segregation of Timestamp to reduce the size, or splitting the data into smaller data sets. But than I will have to use joins to get the complete data set in my dashboard

Re: Data Upload taking too long

Colin-Albert — Thu, 26 May 2016 08:49:54 GMT

Hi Fayez,

Look at implementing an incremental load using concatenate not joins.

Concatenate is the equivalent of a SQL union, and will append additional rows (and columns) into an existing table.

Are you able to split the data in your csv into smaller files, say one for each year or year& month, or one for each region.

There are plenty of posts with more information on incremental loads.

Re: Re: WHat is incremental load ??

http://www.quickintelligence.co.uk/qlikview-incremental-load/