topic Re: REST Connection Data Source - How to handle incremental load? in Connectivity & Data Prep

REST Connection Data Source - How to handle incremental load?

maxlin04 — Thu, 23 Dec 2021 19:00:52 GMT

Can anybody please suggest me a way to handle incremental load in Qliksense while using REST connector?
Present scenario is - We can fetch data from a REST API. The REST api supports pagination by which we can track the delta for a particular time window. We are planning to get this incremental load periodically and update out Qliksense data store without the hassle of loading the whole data set every time. Can anybody please help me here ?

Re: REST Connection Data Source - How to handle incremental load?

dwforest — Fri, 08 Mar 2019 18:20:22 GMT

if the REST source data has a "create/update date", why not just limit it to new records; it would be challenging to get specific dates to be within a certain page range unless your data is very consistent.
The date parameter can be added to the REST in the Query Parameters of the connection dialog.

Re: REST Connection Data Source - How to handle incremental load?

maxlin04 — Tue, 12 Mar 2019 17:03:24 GMT

Hello Dwforest,

Let me explain the scenario a bit more clearly.

We have a data store. It has a REST API using which we can extract data in JSON format. The API has pagination by which we can figure out the new data available in that data store. Pagination is explained below with dummy REST calls.
API call #1 - https://datastore/data/get?start=1&end=5000 <= this REST API call will provide first 5000 data items from data store.
API call #2 - https://datastore/data/get?start=5001&end=6000 <= this REST API all provides the 999 newly added data items.
We need Qlikview to first consume data from url #1 and then every day there will be incremental load by url #2.
Looking forward for expert opinion on what should be the implementation road map for this scenario in Qlikview or QlikSense.