What I do in this sort of case is the following
- read the QVD to get the latest date in the QVD. (Much as you have done)
- Copied the date into a variable (Peek())
- Read the database using the date as a filter. This includes constructing the COMBOKEY
- THEN concatenate load from the QVD using a LOAD * FROM myQVD (qvd) WHERE NOT EXISTS (COMBOKEY);
- Store the resulting table back into the QVD
I think this approach willl work for you as there will be no duplicate COMBOKEYs and the database records will replace the ones that were in the QVD earlier.
These two options produce very different results in your QVD file. Lets look at this first before we consider the number of records to be processed.
- Filtering by "where not exists" in the database would mean that the data that has already loaded into QV would not be updated with the new date. This is the equivalent to an "Add only" approach.
- Filtering by date as described means that the records with new dates will get updated in QV as well - an "Add and Update" approach.
If option 1 is what you need for your solution, then you are going to have to think outside the box. If you have control over the database, and the necessary SQL skills, you could do osomething along these lines:
- add a bit field to the table
- and a script a stored procedure that returns the data for QV.
- The script or procedure would select the records where the bit field = 0, and update the bit field to 1.
- QV reads the records returned by the script/procedure
If option 2 is what you require - then you have the solution - but you will have read more records.
But before you go that far - how many records are we talking about and how frequently do you need to read them? A i million record read does not take very long and may not tax you DB server overmuch (or maybe it will?)