Incremental load Vs. historical view

[saxofonoacn]

Hi,

I created an incremental load for orders where i add new orders to the QVD where Order_KEY does not exist (i.e. only new orders are added). This however causes problems when showing the historical developments of orders (cannot see those orders that did not change over the measured period). Any idea what to do here? Can something be done in the load or is this a front end issue?

Thanks!

.

[swuehl]

Not sure what you want to analyze here, but what about adding the load date (the date where the records were added) to your data? Then you can filter records / orders by last time updated in your app.

[ramoncova06]

can you share your script

[saxofonoacn]

Basically, trying to minimise the data loaded into the app, but for line charts (and line charts only) I want to show the progress of how many orders I had over certain period. The problem is that I am dropping the data at the point of load due to duplication of data - if nothing has changed i don't want to load the data in.

In those line charts I want to say: if the composite key has changed (i.e. an update for an order occurred) take that, otherwise use the previous date's data (i.e. previous load) because nothing has changed for the order.

I am using EXTRACT_DATE already (dimension in the picture above).

Hope this makes sense. Thanks.

[saxofonoacn]

Hi, unfortunately cannot share the script... the picture and text is the best I can do...

[Peter_Cammaert]

You should keep a QVD as a historical cache with all data loaded during the previous reload. When you load an increment the next day, keep the changed records, and add all historical records from the QVD that do not occur already in the list of changed records. Then store the result as the new historical QVD.

Load everything in a single internal table, and you'll have a consistent list of orders (changed and unchanged) without duplicates. That's the basis of performing incremental loads in your script. Mind you, this is not something you do in an UI object.

Peter