Reducing old data volume by aggregation

[galbakal82]

Hi, I have a general question regarding reducing old datasets by aggregating aged data, we plan on having data sets sampled in a certain resolution (dataset every x minutes), and would like to reduce data after certain time (if the data set age is greater than X then aggregate historic dataset to one in X hours) to prevent our app from having enormous amounts of data. Is that possible? Is there a smarter way to cope with this?

Thank you.

[Acrodata]

I think you can do by using QVDs , but Could you elaborate? 

[rwunderlich]

Yes, that's a fairly common technique.  You can aggregate the older data and store it in QVDs.  For example, in a high volume transaction system you might have every transaction for the last 90 days, aggregated values (# of trans, sum(amount)) by Product/Store/Day for 365 days, aggregated values by Month for 5 years.

Aggregated values go in a new field Name like "Daily Amount", "Monthly Amount".

-Rob