I look at the different posts regarding partial Load with the function "Replace" or "Add". But in my case there is something I still can figure out.
Let’s say my script is like this:
FROM ... JAN 2016;
FROM ... FEB 2016;
FROM ... THIS MONTH;
So every time it is loading all the files. But in fact I need only the data from the file “THIS MONTH” to be updated.
Because I have one big table called DATA, if I use REPLACE before LOAD for the file THIS MONTH, the only thing left will be the data from this month. But I want to keep all the data and replacing only the one for this month.
I think what you need is an incremental load, rather than a partial reload. Your incremental load would load the prior months from monthly qvd files. Although you are loading all the data, the qvd loads can be very fast compared to the original data source.
The load for the current month would continually update the qvd for the current month, so that when the month ends, it lands up as a history qvd. Something like:
Let vCurrentMonthFile = Date(Today(), 'MMM yyyy') & '.qvd';
DATA: // replace with the correct load statement for the current period
// replace <your file path> with the correct file path - replaces the current month file every day
STORE DATA into ['<your file path>$(vCurrentMonthFile)'];
// now load the history - make sure to not load the current month
For Each vFile in FileList('<your file path>\*.qvd')
If SubField(vFile, '\', -1) <> vCurrentMonth Then
LOAD * FROM [$(vFile)] (qvd);
When the month ticks over, the previous month's qvd will contain the values for the month and you will start on a new qvd.
Logic will get you from a to b. Imagination will take you everywhere. - A Einstein