incremental reload

[ali_hijazi]

Hello

I'm implementing incremental reload on 2 tables, Header and Details that are associated by a column named INVKEY

The scenario that I have is Insert only on the original datasource

for the very first time I load all data from both tables and store them in qvd files

on the next reload I'm getting the max CreateDate ( a field in which the date at which the record was created is stored) from the QVD file

then I get records from database whose CreateDate >= the max(CreateDate)

then concatenate the retrieved result with the qvd file as follows:

Concatenate(InvoiceHeader)

                              LOAD * from $(vQVDPath)InvoiceHeader.qvd(qvd)

                                        where not(Exists(INVKey));//INVKey is the primary key

                              Store InvoiceHeader into $(vQVDPath)InvoiceHeader.qvd(qvd);

and the Details as follows:

Concatenate(InvoiceDetails)

                              LOAD * from $(vQVDPath)InvoiceDetails.qvd(qvd)

                                        where not(Exists(INVKey));//INVKey is the primary key

                              store InvoiceDetails into $(vQVDPath)InvoiceDetails.qvd(qvd);

suppose the last reload was made on 27/7/2011 morning at 9 and there are records already saved to datasource on 27/7/2011 at 8

then in the after noon new details were inserted in the datasource for existing header records

how can I make sure that these new records entered in the afternoon are saved to the qvd knowing that i put the condition on the details table during concatenat "where not(Exists(INVKey));" to make sure that the records records saved on 27/7/2011 at 8 are not duplicated?