Data Upload taking too long

[Anonymous]

Hi,

I am new to Qlik. I am facing an issue while loading a huge CSV file into Qlik. My approach is first to generate a QVD file. But generation of this file is taking too long aprox 20 mins. The data size is 1.2 GB and my system RAM is 8 Gb.

Please provide me some suggestions which can expedite my process

[vinafidalgo]

The time against the file size seems normal to me.

[Peter_Cammaert]

Well, if text files would have been lightning-fast all along, nobody would have had the crazy idea to invent all sorts of databases. I guess that keeping your data simple & readable comes with a performance disadvantage. A huge one.

If you cannot change the packaging of your data or the nature of the transfer, the best solution is maybe to buy a system with an extremely fast I/O-channel. SSD's aren't that exclusive or expensive anymore. A server has better I/O performance by offering faster interfaces and and in many cases parallelizing disk access...

Peter

[Colin-Albert]

Have you tried using Debug mode with a limited load of say 1000 rows to test that your data load is creating sensible data?

or add a First 1000 command before your load to test the data.

First 1000

load

......

from ...

[Anonymous]

‌yes I did it First 1000.My thinking was can I implement some kind of best practices like segregation of Timestamp to reduce the size, or splitting the data into smaller data sets. But than I will have to use joins to get the complete data set in my dashboard

[Colin-Albert]

Hi Fayez,

Look at implementing an incremental load using concatenate not joins.

Concatenate is the equivalent of a SQL union, and will append additional rows (and columns) into an existing table.

Are you able to split the data in your csv into smaller files, say one for each year or year& month, or one for each region.

There are plenty of posts with more information on incremental loads.

Re: Re: WHat is incremental load ??

http://www.quickintelligence.co.uk/qlikview-incremental-load/