I have a Windows Server 2003 Enterprise x64 Edition Service Pack 2 with thefollowing settings: 1) Intel XEON 1.60GHz 2) 16 GB RAM
I’ve install Qlikview 64 bit SR3 10.00.9055.7 Edition (x64) and I'm loadingaCSV file (10 Gb) with 100 million records distributed in 8 columns, whoserecord maximum length can be 50 characters and it stops at 77.5 million recordsafter 15hrs of making the initial load.
The competition’s tool does load in 4 hrs, the intention is to improve this timeto give a solution as a tool for BI.
Are you loading the file directly or via driver? Is it only one file or multiple files in a loop?
Anyway, note that you can LOAD the CSV file into QlikView, then STORE into a QVD file and then use this QVD file for anything, and, just in case, load the non-existent records from the CSV storing always in the QVD. Have you tried loading from QVD?
If loading half the volume can be done in 15 minutes, but the full volume takes 15 hours then stops, something is definitely wrong. Typically I'd say "you don't have enough memory", though it looks like you do on the surface. Have you watched your memory consumption during the 15 hour load to see if it maxes out?
Thank you Guys, I already finished with it, I load the csv file in QVW file. It works perfect, I also generate a QVD file. The compression from the 10 Gb file at the end is 6 Gb. For certain reason, I couldn't load the QVD file. So I decided to load the QVW file as Binary, it takes me just 2 minutes.
Go to the script editor (Ctrl + E) and use the "Table File" button in the bottom part of the dialog and follow the wizzard that will show up. That will help you loading the CSV file, setting the separator (or fixed positions, if any).