Incremental Load in Qlik Sense

jhstrange — Wed, 30 Jan 2019 17:07:16 GMT

Hello all,

We have a "last 30-days" API feed that comes in daily.The historical data would be loaded into a QVD as a one-time activity and we get "last 30 days" feed every day.

On Jan 27th,2019 the API feed would give us data for the time period Dec 28th,2018 to Jan 27th,2019.On Jan 28th,it would be Dec 29th to Jan 28th data.How do we make sure we are not missing out on Dec 28th data and that we are permanently storing the preceding day's data(28th Dec in this case) along with the historical data and current data at one place?Let us assume that the historical data is available from Jan 1,2016 to Dec 27th,2018 for time being.

Any help would be greatly appreciated!

Re: Incremental Load in Qlik Sense

Gysbert_Wassenaar — Wed, 30 Jan 2019 17:14:44 GMT

Usually this is done with some variation of this:

1. Load all the data from the last-30-days feed

2. Load only the data from the historical qvd with dates that aren't already loaded in step 1

3. Store the complete set from step 1 and 2 into the historical qvd.

Data:
LOAD 
    ...some fields...,
    DateField,
    ...some more fields...
FROM
    ...last_30_days_source...;

CONCATENATE (Data)

LOAD 
    * 
FROM
     [LIB://QVDStore\HistoricalQVD.qvd] (qvd)
WHERE
    NOT Exists(DateField)
    ;


STORE Data INTO [LIB://QVDStore\HistoricalQVD.qvd] (qvd);

topic Re: Incremental Load in Qlik Sense in App Development

Incremental Load in Qlik Sense

Re: Incremental Load in Qlik Sense