Incremental in Big Data

Faeze_M — Wed, 08 Jan 2025 06:29:20 GMT

Hi everyone,

I have a question regarding the Incremental Load mechanism in QlikView. As you know, implementing Incremental Load typically requires a field like Last Modified or its equivalent in the QVD to identify which records have changed. This allows us to load only the new or updated data and merge it with the existing dataset.

However, in cases where such a field does not exist in the QVD, it seems Incremental or Partial Load isn't feasible. Instead, all data—whether changed or unchanged—gets added to the target table during each load, effectively concatenating the new data with the old.

My question is:
Is there any way to handle this scenario? Specifically, can we implement Incremental Load without having a field that tracks changes? I’d really appreciate it if you could share any solutions or experiences related to this situation.

Re: Incremental in Big Data

pallavi_96 — Wed, 08 Jan 2025 08:00:55 GMT

Hi @Faeze_M

1) If your data source or QVD has log files or any time-based records (like a system-generated timestamp for data creation), you can use those timestamps to perform Incremental Load. Even if the source data doesn't have a "Last Modified" field, a creation timestamp (like CreatedDate or a similar field) could allow you to track newly added records

For example, if you're pulling data from a database, check if there's a "CreatedDate" or "Timestamp" field

2) If the data you're working with has a versioning system (e.g., different versions of records are stored and can be tracked by a version number), you can compare the version numbers between loads to identify new or updated data.

topic Re: Incremental in Big Data in Featured Members

Incremental in Big Data

Re: Incremental in Big Data