Incremental stored. A practical case

[alexandernatale]

Good morning everyone,

I was trying to create an incremental data load on qliksense saas. It would seem that the script works but I would like to be sure that I understand the mechanism correctly.

This is the script...

LIB CONNECT TO 'Earthquake INGV';

RestConnectorMasterTable:

SQL SELECT 

"#EventID",

"Time",

"Latitude",

"Longitude",

"Depth/Km",

"Author",

"Catalog",

"Contributor",

"ContributorID",

"MagType",

"Magnitude",

"MagAuthor",

"EventLocationName",

"EventType"

FROM CSV (header on, delimiter "|", quote """") "CSV_source";

[ingv]:

LOAD [#EventID] AS Id,

[Time],

[Latitude],

[Longitude],

[Depth/Km],

[Author],

[Catalog],

[Contributor],

[ContributorID],

[MagType],

[Magnitude],

[MagAuthor],

[EventLocationName],

[EventType]

RESIDENT RestConnectorMasterTable;

DROP TABLE RestConnectorMasterTable;

[ingv]:
LOAD * FROM [lib://Google_Drive - xxxxxxxxxxxxxxx@gmail.com/root/ingv.qvd] (qvd)
WHERE NOT EXISTS (Id);
store ingv into [lib://Google_Drive - xxxxxxxxxxxxxxx@gmail.com/root/ingv.qvd];

As far as I understand:
1. the rest connector loads a temporary table RestConnectorMasterTable to store the query result;
2. I generate a table on qliksense called ingv where I'm going to store the result of the query; the RestConnectorMasterTable table is deleted;
3. upload to ingv all the fields present in the .qvd file whose id does not already exist in the ingv table;
4. I store the "final" ingv table in the same .qvd file overwriting the results.

It's correct?

[Benoit_C · Accepted solution]

Hello @alexandernatale,

Yes you perfectly understood the script.

This is a typical pattern for incremental data loads in Qlik Sense, where you load new data from a source, compare it with the existing data, and add only the new records. By doing this, you can reduce the time and resources required for data loads and ensure that your Qlik Sense application has the latest data available.

Regards,

Benoit