I saw your data set, I have few queries 1.Everyday you will get years of years date value? 2.If yes, will their be any changes to the old records? 3.If you dont have update date column or lastest change columns try to base the incremental logic on primary key
Incremental loads can be done in multiple ways depending on how data is updated and the needs of the business.
Here is a simple approach fetching all new rows based on the Day field.
I assume you have an inc_data.qvd with previous loaded data.
inc_data: Load * From inc_data.qvd (qvd);
TmpMaxDay: LOAD Max(Day) as MaxDay; LOAD FieldValue('Day', recno()) as Day AUTOGENERATE FieldValueCount('Day'); Let vMaxDay = peek('MaxDay',-1,'TmpMaxDay')
Concatenate (inc_data) Load * from DataSource Where Day > $(vMaxDay);