Skip to main content
Announcements
Accelerate Your Success: Fuel your data and AI journey with the right services, delivered by our experts. Learn More
cancel
Showing results for 
Search instead for 
Did you mean: 
Sugarp
Contributor II
Contributor II

Incremental load

I have 1 year of data.data gets added every week. I want to use incremental load so that I don’t have to load the whole data again and agian. I have date field and also counter filed to identify new data . How do I do that. Please help

Labels (2)
1 Reply
rubenmarin

Hi, ther are many post explaining how to do that, some examples:

https://help.qlik.com/en-US/sense/February2024/Subsystems/Hub/Content/Sense_Hub/LoadData/use-QVD-fil...

https://www.quickintelligence.co.uk/qlikview-incremental-load/

https://www.youtube.com/watch?v=GLO2UteiHQ8&ab_channel=QlikHelp

Basically it starts retriving the higher value you have already loaded, this can be done by reading the already created qvd, or using a variable that stores the last loaded time or ID.

Then load data from the database vales higher to those values, and add from the qvd the values previously loaded. 

Then the table loaded with the new and old values is stored again as qvd.

This makes the qvd growing bigger, adding only the new/modified records.

If there could be also deleted records, it complicates a bit because you need to load all the rows from db to check if any of the existing rows on the qvd has to be excluded in the next incremental load, the approch is also explained in the first link.