How to Implement Incremental Load in 2nd and 3rd Tiers of Qlik Sense Three-Tier Architecture

[Purushothaman]

Hi Qlik Experts,

I’m currently working with a three-tier architecture in Qlik Sense and I’d appreciate any guidance on how to set up incremental load in both the 2nd tier (Data Model) and the 3rd tier (Dashboard) layers.

Architecture Overview:

• 1st Tier (Extraction):

  • I have already set up incremental refresh in the extraction layer. Data is loaded incrementally from the source and stored in QVDs. This part is working fine.

• 2nd Tier (Data Model):

  • Data in this layer is loaded from QVDs created in the extraction layer. This is where I’m looking to set up incremental loading for optimized performance and reduced load times.

  Challenge:
  How can I implement incremental logic in this tier to avoid loading the full dataset every time? I want to ensure only new or changed data from the QVDs are loaded into the data model.

• 3rd Tier (Dashboard):

  • The dashboard layer uses a binary load to load the data model from the 2nd tier.

  Challenge:
  I know the binary load takes the entire data model into the dashboard app. Is there a way to optimize or implement an incremental approach here? Or would I need to rely on partial reloads or some other strategy?
  
  
  Any suggestions or best practices for setting up incremental loading across these tiers would be greatly appreciated!

  Thank you for your time and help!

[marcus_sommer]

It's for the second/third tier mainly the same approach as by the first layer - checking the historic and current data against any existing key-field and/or against extracted variables, for example max. key-values or timestamps or similar.

Quite helpful could also be to slice the data for any categories especially period-information in qvd's and to include them also within the file-name. Afterwards these information could be queried within a filelist-loop and the incremental approach might be reduced to load everything to the current period - n and all the others comes from the current stack.

Such methods might be also applied after a binary load in the last layer.

[Purushothaman]

@Dataintellinalytics 

May I have your thoughts in this approach. If possible. Please.

Thank you!