topic Re: Increment Load Implementation without having incremental logic in App Development

Increment Load Implementation without having incremental logic

Ruchika_Thakral — Tue, 18 Mar 2025 08:07:50 GMT

My fact table lacks both a Modification Date & Time field and a Unique ID. How can I implement Incremental Load in this scenario?

Re: Increment Load Implementation without having incremental logic

marcus_sommer — Tue, 18 Mar 2025 13:11:14 GMT

You may use a normal date-field and regarding the dates from today() - n as new and the elder ones as already existing and unchanged. It may not be perfect but will mostly serve the purpose - and it may only be the daily task and once a week/month in any idle hours a full-load is performed to fetch the neglected exceptions.

Re: Increment Load Implementation without having incremental logic

morgankejerhag — Tue, 18 Mar 2025 13:15:26 GMT

If lines can be added and modified (and even removed) then you can't create that in the script. Look at Qlik Talend Cloud and CDC technology to stream changes.

Re: Increment Load Implementation without having incremental logic

Or — Tue, 18 Mar 2025 13:25:31 GMT

"My table lacks the requirements for incremental loading, how can I load incrementally?"

You can't, generally. You have to ask yourself (or your client) what logic can be used to recognize changes that need to be loaded, and how to recognize that the row in question needs to be changed, and if such a logic is available, that's your answer.