I pull in 6 months of data for a call center which equates to about 40 million records. There are no updates to the call data. I must perform a number of transforms and intervalmatches to get the data in the format I need.
Since the past data does not change, is it possible to pull in the already transformed and interval matched qvd and add to it.
Something like:
Load transformed qvd. On the new load, pull in a day prior to the end of the existing qvd. Perform all the transormations and intervalmatch on the new incremental values and merge with the existing.
The issue would be to drop the ending partial intervals, and replace with the now full and new. Also, you would not want to merge in the partial intervals at the beginning of your new load. You would have those from the previous.
I hope what I am saying makes sense.
This is a daily task that can take 30min to basically rebuild over 6 months every time when only adding a new day.
Thanks
