topic Re: Incremental Load in App Development

Incremental Load

l26b — Wed, 16 Jul 2025 16:01:32 GMT

Hi,

I have a really complicated query that I am running now in qlik for a long period of time and extracting data and store it into qvd.

I want to move to incremental load but the script I tried was deleting data from the past 2 days.

Any idea how to make this work? I can send full extract app in private

Re: Incremental Load

edwin — Thu, 17 Jul 2025 13:25:49 GMT

i can give it a try

Re: Incremental Load

Kaushik2020 — Wed, 06 Aug 2025 07:32:54 GMT

Hi @l26b Please share the query. I can give a try. I hope there is a date field already there as part of the Table.

I have done one where i will check if there is any existing qvd there with yesterdays date. if YES then load only todays data and concatenate with the existing QVD.

if none, it will create a new QVD with complete data.

This helped in cases where there were billions of records in the table.

Re: Incremental Load

stevedark — Thu, 07 Aug 2025 15:53:01 GMT

Hi @l26b

Without seeing your app I can't suggest the best way to do an incremental load. A lot depends on whether you have a unique ID and/or update date field in the table.

This old blog post has some suggestions for different types of data. It was written about QlikView, but the script is entirely identical in Sense:

https://www.quickintelligence.co.uk/qlikview-incremental-load/

Hope that helps.

Steve

Re: Incremental Load

savioricardog — Thu, 07 Aug 2025 17:47:38 GMT

Hi @l26b Could you share your script and a small sample of your data? It’ll be easier to help you.

Re: Incremental Load

l26b — Fri, 24 Oct 2025 08:43:09 GMT

Hi,

I can send it in private message

Re: Incremental Load

l26b — Fri, 24 Oct 2025 08:43:24 GMT

I can send it in private

Re: Incremental Load

l26b — Fri, 24 Oct 2025 08:43:40 GMT

I can send it in private

Re: Incremental Load

l26b — Fri, 24 Oct 2025 08:48:37 GMT

Re: Incremental Load

l26b — Fri, 24 Oct 2025 08:49:10 GMT

hi @savioricardog attached the load script

thank you!!

Re: Incremental Load

l26b — Fri, 24 Oct 2025 08:49:39 GMT

script is attached

Re: Incremental Load

marcus_sommer — Fri, 24 Oct 2025 14:02:03 GMT

Because of the fact that the sql contains heavily nested sub-queries an appropriate incremental control-logic might be placed n times. I suggest to start with an optimizing of the sql - for example by removing the nested queries and simplifying as much as possible.

And maybe you ends with n quite pure raw-data queries to Qlik and applying the wanted transformations there. In each case you need to understand exactly what the sql is doing to be able to apply anything on top of it.

Re: Incremental Load

Kaushik2020 — Sun, 26 Oct 2025 12:19:24 GMT

My suggestion will be to create a stored procedure using the script which you have shared above into a table in SQL. Further use incremental load based on the progress_date column/Data field from the table to describe the QVD generation. this way your load will be quick and you will load only the new records as part of QVD creation. Happy to help you with optimization in SQL as well as Qlik Sense.