I want to load data from MS SQL Database(208 million records) to a QVD File. I am trying to load the data in incremental way, ie if the file doesnt exists then i should start loading data from the beginning. If data was previously loaded from the database to a QVD file then it should load only those records which are new in the database.
I want to create only 1 script which does the job for me. I don't want to load first 100 data in to a QVD file and then load other data in incremental fashion. All this should done in a shot (i.e Script) when i execute the script. Script should be capable enough to detect whether there is previously data pulled or not and then based on that it should start loading the data. QVD FileName will be predefined.
I want to know whether it is possible to load such huge data to a QVD File.
If yes then how? want a sample script which should do the job for me.
I have a int field which is unique and trying to get this thing done based on that.
Db structure : ID int,FirstName nvarchar(50), LastName nvarchar(50) ,dtCreatedDate datetime, IsActive bit
My Logic is
Step 1 : Check whether a qvd file exists
Step 2 : if no, get the total no of records from db and store it in a variable totalrowcount. second initialize variable qvdrowcount as 0.
Step 3: use a do.. loop which should will first pull 100000 records from database to my qvd file. Then increment the qvdrowcount variable by 100000. Get the Last ID NO from the QVD file.
STep 4: check whether qvdrowcount < totaldbrowcount. If yes then pull next set of 100000 records where ID NO Greater than the lastIDNO from qvd file and saves it to the same qvd file.
Step 5: this continues till my while expression "qvdrowcount < totaldbrowcount" returns false and the loop breaks.
Step 6 : Incase if data was pulled previously in a qvd file then, get the recordcount from qvd file, get the max ID NO from the qvd file and then follow the steps from 3 to 5.
Kindly suggest whether this approach is a optimal solution or not. If No, then kindly guide me in achieving this task using QLikView.
Thanks in advance for the help.