Incremental load

[#1]

I was asked a question in one of internal assessments, which I was not very sure of

When is Incremental load beneficial?

- When the number of new transactions more or

- When there are frequent changes in the data

Please share your views. Will be really helpful.

[swuehl]

You can assess these questions better if you create some scenarios and try to determine what happens using an incremental load (I assume you know how the incremental load works, do you?).

For example, scenario A may look like

100 initial records with primary key ID

100 records added per day (indicated by new IDs and MODIFIEDAT field)

scenario B may look like

100 initial records with primary key ID

same 100 record IDs changed each day (indicated by a MODIFIEDAT field)

what do you think happens when using an incremental LOAD?

(if in doubt: https://help.qlik.com/en-US/sense/3.2/Subsystems/Hub/Content/LoadData/use-QVD-files-incremental-load...)

What do you think, which scenario is handled efficiently by a daily incremental load?

[#3]

Hi Stefan

Thanks for your response. As per my understanding from the scenarios it is good to use incremental load when the number of new transactions are more.

Do you agree with me?

[swuehl]

Yes, I do.

These are of course extreme scenarios, but if all records change each day, there is no benefit in using the incremental load concept, in fact there is even an additional overhead compared to a plain full load.

Rule of thumb, the more records you can get from your historic QVD compared to new / updated records from the data source, the better the incremental load will work (in terms of performance, assuming the QVD access is much faster than your data source access).

[#5]

I think it is very very scenario specific. It's difficult to choose between the two options.