Qlik Community

Ask a Question

QlikView App Dev

Discussion Board for collaboration related to QlikView App Development.

Announcements
May 18th 10AM ET, Live Chat, bring your QlikView questions. REGISTER
cancel
Showing results for 
Search instead for 
Did you mean: 
Not applicable

Volume initial load performance

I have a Windows Server 2003 Enterprise x64 Edition Service Pack 2 with thefollowing settings:
1) Intel XEON 1.60GHz
2) 16 GB RAM

I’ve install Qlikview 64 bit SR3 10.00.9055.7 Edition (x64) and I'm loadingaCSV file (10 Gb) with 100 million records distributed in 8 columns, whoserecord maximum length can be 50 characters and it stops at 77.5 million recordsafter 15hrs of making the initial load.


The competition’s tool does load in 4 hrs, the intention is to improve this timeto give a solution as a tool for BI.

Does anyone know how to optimize the load volume?

Greetings!

7 Replies
Miguel_Angel_Baeyens

Hello Jose,

Are you loading the file directly or via driver? Is it only one file or multiple files in a loop?

Anyway, note that you can LOAD the CSV file into QlikView, then STORE into a QVD file and then use this QVD file for anything, and, just in case, load the non-existent records from the CSV storing always in the QVD. Have you tried loading from QVD?

Hope that helps.

Miguel Angel Baeyens

BI Consultant

Comex Grupo Ibérica

Not applicable
Author

Hi Miguel Angel

The load is direct

There is only one file

I've already store the load in QVD file

What I've done is split the file in 2 QVDs files, it works perfect, cause it finished in 15 min each file. Now I'm trying to concatenate both files in one QVD file, I hope it works.

Do you have another suggestion?

Thanks in advance.

johnw
MVP
MVP

If loading half the volume can be done in 15 minutes, but the full volume takes 15 hours then stops, something is definitely wrong.  Typically I'd say "you don't have enough memory", though it looks like you do on the surface.  Have you watched your memory consumption during the 15 hour load to see if it maxes out?

Not applicable
Author

Thank you Guys, I already finished with it, I load the csv file in QVW file. It works perfect, I also generate a QVD file. The compression from the 10 Gb file at the end is 6 Gb. For certain reason, I couldn't load the QVD file. So I decided to load the QVW file as Binary, it takes me just 2 minutes.

Thank you for your suggestions.

Not applicable
Author

I'm having trouble w/ the syntax to create a qvd from a csv.  (I am fine with select * from a database).  Can you give me an example? 

Miguel_Angel_Baeyens

Hi Anna,

Go to the script editor (Ctrl + E) and use the "Table File" button in the bottom part of the dialog and follow the wizzard that will show up. That will help you loading the CSV file, setting the separator (or fixed positions, if any).

Hope that helps.

Miguel Angel Baeyens

BI Consultant

Comex Grupo Ibérica

Not applicable
Author

all set!  got busy, hence my delayed response but did want to say thanks!