Incremental load - will it skip rows?

[sifatnabil]

Hi all,

I've implemented incremental load with date restriction, the algo is like this:

Check if QVD exists,

if yes,

     load from QVD

else

     load from database

store into QVD

concatenate

load from database

where date>=today's date

//End

If it loads at 10am on Aug 22, and again at 11pm on Aug 22, and if new rows have been added to the database between 10am and 11am, will the 11am refresh load the correct data? i.e. will the 11am refresh load all rows starting from Aug 22 00:00:00 and overwrite the 10am refresh? Because this would be correct then. Thanks.

[antoniotiman]

Hi,

I think

Check if QVD exists,

if yes

load from QVD;

Load from database

where date >= today;      (or where date >= last date loaded)

Store into QVD;

Regards

[sifatnabil · Accepted solution]

Thanks, I used primary keys and LOAD DISTINCT to ensure that duplicate rows were not created during concatenation.