Incremental load

vadim_grab — Sun, 19 Aug 2018 09:24:49 GMT

Hi, All.

I'm studying examples scenario of incremental data loading to use best practice in current project.

Most common scenarion is Insert, Update and Delete.

One of the examples below (from book Mastering Qlik Sense Packt March 2018).

I have a question -- What will be if COUNTRY don't have a new data?

The table COUNTRY will not be created, so the data will not be loaded at all.

This question also actuals for most of the incremental load examples that can be found in Qlik community.

Maybe I'm missing something.

Many thanks for your comments and advices!

Best regards,

Vadim.

//I check if the file exists. If it doesn't I do a full load, if it does I perform the incremental load.

IF IsNull(QvdCreateTime('C:\myQVD\Countries.qvd')) THEN

TRACE Performing full load of data source;

Countries:

SELECT * FROM COUNTRY;

STORE Countries INTO C:\myQVD\Countries.qvd (qvd);

ELSE

TRACE File exists. Performing incremental load;

maxDate:

LOAD max(LastModifiedDate) AS MaxModifiedDate

FROM c:\myQVD\Countries.qvd (qvd);

TRACE Storing the most current modified date into a variable.;

LET vMaxModifiedDate = peek(MaxModifiedDate);

TRACE Loading new records from the source.;

Countries:

SELECT * FROM COUNTRY

WHERE LastModifiedDate > $(vMaxModifiedDate) // My question - What will be if COUNTRY don't have a new data

;

TRACE Appending stored records to new ones.;

Concatenate

LOAD * FROM c:\myQVD\Countries.qvd (Qvd)

WHERE Not EXISTS(%Id);

TRACE We join the final table with all the current keys in the source to remove the delete records.;

INNER JOIN

SQL SELECT %Id FROM COUNTRY;

TRACE Storing modified qvd file.;

STORE Countries INTO c:\myQVD\Countries.qvd (Qvd);

ENDIF

Re: Incremental load

andrey_krylov — Sun, 19 Aug 2018 09:41:04 GMT

In this case you get the table with all fields and no rows but after concotenating

Concatenate

LOAD * FROM c:\myQVD\Countries.qvd (Qvd)

WHERE Not EXISTS(%Id);

you get the full table

Re: Incremental load

mohan_1105 — Sun, 19 Aug 2018 15:58:14 GMT

Hi Vadim,

If I understood correctly, your query will not fetch any new records and the qvd contains only the records fetched in the previously loaded data.

Re: Incremental load

marcus_sommer — Tue, 21 Aug 2018 11:27:23 GMT

Something like:

t: load F1, F2, ... from Source where 1=2;

will lead to an empty table - all fields are available but no records. And this meant also that you could concatenate the qvd to such a table.

- Marcus

topic Incremental load in QlikView

Incremental load

Re: Incremental load

Re: Incremental load

Re: Incremental load