Qlik Sense - Rest API (Json) + Pagination + Incremental load (new and update)
Hey, I'm thinking how to apply incrementation (insert and update) when downloading data to a QVD file via REST API (JSON file) and pagination to pages with 100 records (required) at the same time.
The data retrieved is all the data since the last change.
I have done the data download with pagination at the moment. And now I wonder where to put the incrementation
let vStartAt = 0; set vPageSize = 100;
Do While vStartAt <= vTotalFetched
let vUrl = 'https://{URLAddress}/search?modifiedOn=2023-09-01T00:00:00.000Z&size=' & $(vPageSize) & '&page=' & $(vStartAt) & 'sort=id';
RestConnectorMasterTable:
SQL SELECT
(...)
WITH CONNECTION (
URL "$(vUrl)",
HTTPHEADER "Authorization" "Bearer $(vAccessTokenValue)"
);
Store RestConnectorMasterTable into [lib://StageFile/RestConnectorMasterTable.qvd](qvd);
(...)
[orderContact]:
LOAD [email],
[customerNo],
[firstName],
[lastName],
[__FK_orderContact] AS [__KEY_content]
RESIDENT RestConnectorMasterTable
WHERE NOT IsNull([__FK_orderContact]);
(...)
DROP TABLE RestConnectorMasterTable;
Let vStartAt=vStartAt+1;
Loop