The INSERT & UPDATE scenario also takes new data from the source but it also pulls in updated records. Additional precautions need to be taken in order to avoid duplicate records. During the load from the QVD, exclude records where there is a match on the primary key. This will ensure that the updated records will not be duplicated.
Load all NEW and UPDATED data from the data source
Concatenate this data with a load of only the missing records from the QVD file
Store the entire table out to the QVD file
DataTable: SQL SELECT PrimaryKey, A, B, C FROM Table1 WHERE ModifyDate >= $(vDate);
LOAD PrimaryKey, A, B, C FROM ABC.qvd WHERE NOT exists (PrimaryKey);
Hello, With Partial Reload you can't update the already loaded table data. Either you can replace whole table or add the data. If your latest file have the full data set, simply use partial reload to replace the table.
Otherwise, do full reload with full incremental load.
You can also do incremental load on csv. But I am not sure it will give the better performance.