Incremental Load

[l26b]

Hi, 

I have a really complicated query that I am running now in qlik for a long period of time and extracting data and store it into qvd. 

I want to move to incremental load but the script I tried was deleting data from the past 2 days. 

Any idea how to make this work? I can send full extract app in private

[edwin]

i can give it a try

[Kaushik2020]

Hi @l26b Please share the query. I can give a try. I hope there is a date field already there as part of the Table. 

I have done one where i will check if there is any existing qvd there with yesterdays date. if YES then load only todays data and concatenate with the existing QVD.

if none, it will create a new QVD with complete data. 

This helped in cases where there were billions of records in the table. 

[stevedark]

Hi @l26b 

Without seeing your app I can't suggest the best way to do an incremental load. A lot depends on whether you have a unique ID and/or update date field in the table.

This old blog post has some suggestions for different types of data. It was written about QlikView, but the script is entirely identical in Sense:

https://www.quickintelligence.co.uk/qlikview-incremental-load/

Hope that helps.

Steve

[savioricardog]

Hi @l26b Could you share your script and a small sample of your data? It’ll be easier to help you.