You may consider moving the files to a different folder after loading and storing to your QVD.
Or create a loop of the files and only load the newest files
FOR Each File in filelist (Root&' \*.' &Ext)
Let vCheck = ... // Parse the date from file name and compare to latest Date from QVD
IF vCheck THEN
You can use a variable with the actual date for loan only the file of this date:
Store Table into ..\QVD\ActualQVD.qvd;
and after that you can do a incremental load
Let me know if this help you.
Thank you! I pursued swuehl second option. Script execution is significantly faster, but I must admit the script takes somewhat longer than I anticipated to pull the file names from my network drive.
My code for reference:
//Loading Data from QVD
FROM [c:/blah.qvd] (qvd);
//Find Last Modified Date
LOAD Max(Modified_Date) as MaxDate
//Store Last Modified Date to a Variable
LET Last_Updated_Date = peek('MaxDate',0,'Last_Updated_Date');
FOR Each vFile in FileList('C:\ABC\TOR*.csv')
LET vfilenamedate = Date#(mid('$(vFile)',23,8), 'YYYYMMDD');
IF (vfilenamedate > Last_Updated_Date) then
// Replace Old QVD file
IF (vStore = 'Yes' ) then
Store [DATA_TABLE] into c:/blah.qvd (qvd);