topic Incremental load - will it skip rows? in QlikView

Incremental load - will it skip rows?

sifatnabil — Fri, 22 Aug 2014 07:08:17 GMT

Hi all,

I've implemented incremental load with date restriction, the algo is like this:

Check if QVD exists,

if yes,

load from QVD

else

load from database

store into QVD

concatenate

load from database

where date>=today's date

//End

If it loads at 10am on Aug 22, and again at 11pm on Aug 22, and if new rows have been added to the database between 10am and 11am, will the 11am refresh load the correct data? i.e. will the 11am refresh load all rows starting from Aug 22 00:00:00 and overwrite the 10am refresh? Because this would be correct then. Thanks.

Re: Incremental load - will it skip rows?

antoniotiman — Fri, 22 Aug 2014 07:32:27 GMT

Hi,

I think

Check if QVD exists,

if yes

load from QVD;

Load from database

where date >= today; (or where date >= last date loaded)

Store into QVD;

Regards

Re: Incremental load - will it skip rows?

sifatnabil — Thu, 28 Aug 2014 09:53:40 GMT

Thanks, I used primary keys and LOAD DISTINCT to ensure that duplicate rows were not created during concatenation.