Skip to main content
Announcements
Have questions about Qlik Connect? Join us live on April 10th, at 11 AM ET: SIGN UP NOW
cancel
Showing results for 
Search instead for 
Did you mean: 
rahulpawarb
Specialist III
Specialist III

What is best reload strategy for Incremental load?

Hello All,

Trust that you all are doing well!

At present I have two QVW files to fulfill the incremental load requirement:

  1. History.qvw – It will fetch entire data from the source system and dump it into QVD file.
  2. Delta.qvw – It will fetch only the changed data during last 30 days and dump it into QVD file.

What could be the best reload strategy? How many tasks we have create? What would be the ideal time of task execution (Ideally, it should be during offline hours)? What could be the frequency of each task?

Looking forward to get more inputs on this.

Regards!

Rahul Pawar

4 Replies
rahulpawarb
Specialist III
Specialist III
Author

stalwar1

gwassenaar

dvqlikview

mto

devarasu07

Your valuable inputs on this will be of help.

Regards!

Rahul Pawar

Gysbert_Wassenaar

Could be done with one app and thus one task. The task would run once a day and retrieve the data of that last 30 days and store that data in a qvd file. Next it would load all the data from the historical data qvd, but exclude any data that is already loaded from the last 30 days. That historical data is appended to the data of the last 30 days and the result is stored in the historical qvd.


talk is cheap, supply exceeds demand
rahulpawarb
Specialist III
Specialist III
Author

Thank you Gysbert for valuable inputs!

Here I have 1 question: How we are going to access/maintain the history data? For example, If I extracted history data on 1st Jan 2018 and stored it in a qvd. Post this I am using Delta load to fetch changed data in last 30 days. This will work fine till 31st Jan 2018. If I follow same reload strategy on 5th Feb 2018 then delta load will fetch data from 6th Jan 2018 to 5th Feb 2018 and history have data till 1st Jan 2018. What will happen to data between 1st Jan 2018 and 6th Jan 2018? How we are going to get this data into application?

Please offer your valuable inputs on this.

P.S.: Loading History data on daily basis is not feasible solution.

Regards!

Rahul Pawar

swuehl
MVP
MVP

You are basically not using a fixed date range for the incremental load, but store and retrieve the last timestamp of your last reload from source (i.e. your historic qvd). Then use this as filter when loading the delta records from source in the next reload and append to your historic qvd.

Just refer to the many examples here in the forum, the online HELP or just use Rob Wunderlich's QlikView components QVC library.