Qlik Community

App Development

Discussion board where members can learn more about Qlik Sense App Development and Usage.

Announcements
Don't miss the upcoming Q&A with Qlik session on Qlik Application Automation on November 16th! REGISTER NOW
cancel
Showing results for 
Search instead for 
Did you mean: 
Qlik_Enthu
Creator II
Creator II

Points to consider before Implementing Incremental Load (Insert only) to an existing Qliksense application that runs every hour!

Hello Folks!

I am trying to implement incremental load (insert only) to my existing app in Qliksense. My app refreshes every hour.

The data volume is huge and we wanted to optimize the data load by this activity.

I assume, since the data is getting refreshes every hour, I can directly add the logic for incremental load and can put the code in production without much thinking. So that, the new records will be added in the next hourly run during the deployment process.

Kindly correct me if my understanding is correct or not! 

Should there be any other considerations or things to take care w.r.t the data integrity aspect?

TIA!

Regards...

1 Reply
Dalton_Ruer
Employee
Employee

Not sure what you mean by HUGE so I will give two answers:

1. Incremental loading is definitely easy to implement for those where it is truly INSERT ONLY.  Simply use CONCATENATE to add the new data to the existing data. 

2. Many with really HUGE volumes of data don't keep it all together. Instead they keep it Monthly, Weekly, Daily type QVD files. Their front end applications then load whatever is needed. 

3. For those reading this who have to worry about Update and Delete transactions be sure and check out the new MERGE functionality that is available: https://help.qlik.com/en-US/sense/February2021/Subsystems/Hub/Content/Sense_Hub/Scripting/ScriptPref...