Storing Data in Daily basis.

[#1]

Hi Everyone,

     I am developing an application that loads data from the online website in an daily basis.

     I want to store the historical data that means yesterday's data and I will see that in future.

     To be very clear, While loading, I have specified a column called date which is the current date.

     My Question is what if I want to see the yesterday's data. How can store that in a qvd?

     Currently, The loads replaces previous day data and stores the data available today.

     I heard that there is concept called Incremental Load. Will this be helpful to me?

     If yes, kindly elaborate on what is incremental load and how to do that?

Regards

Rajan

[swuehl]

Yes, I think incremental reload would be a possible solution.

Basically, an incremenal reload will only load new or updated data from your source table, in your case, I would assume just retrieving today's data. Then the historical data is appended from qvd and the new complete table is stored back into qvd.

So you start with no qvd at the first day, you retrieve your data from web source and store it to a qvd. Second Day, you retrieve data for the second day, append qvd for the first day data and store both day's data into qvd. Third day, you retrieve data for the third ('today') day from your web source, append qvd with data for second and first day and store all three day's data into qvd. And so on.

It's a bit more complicated to allow for new / updated / deleted data (and if you don't know what the new data is, you need to keep an modification date or recID, to compare source data and qvd data).

You will find an excellent example in the QV cookbook, available from Rob Wunderlichs download site.

http://robwunderlich.com/downloads/

Hope this helps,

Stefan