Points to consider before Implementing Incremental Load (Insert only) to an existing Qliksense application that runs every hour!

[Qlik_Enthu]

Hello Folks!

I am trying to implement incremental load (insert only) to my existing app in Qliksense. My app refreshes every hour.

The data volume is huge and we wanted to optimize the data load by this activity.

I assume, since the data is getting refreshes every hour, I can directly add the logic for incremental load and can put the code in production without much thinking. So that, the new records will be added in the next hourly run during the deployment process.

Kindly correct me if my understanding is correct or not! 

Should there be any other considerations or things to take care w.r.t the data integrity aspect?

TIA!

Regards...

[Dalton_Ruer]

Not sure what you mean by HUGE so I will give two answers:

1. Incremental loading is definitely easy to implement for those where it is truly INSERT ONLY.  Simply use CONCATENATE to add the new data to the existing data. 

2. Many with really HUGE volumes of data don't keep it all together. Instead they keep it Monthly, Weekly, Daily type QVD files. Their front end applications then load whatever is needed. 

3. For those reading this who have to worry about Update and Delete transactions be sure and check out the new MERGE functionality that is available: https://help.qlik.com/en-US/sense/February2021/Subsystems/Hub/Content/Sense_Hub/Scripting/ScriptPref...