Incremental load from multiple QVD files

[#1]

Hi all,

Our QV module reads data from multiple QVD files stores in a specific directory.

Each day new file is being added to the same directory.

How can we make the load of the qvw to take only the newly added files (since last update)?

Currently, we open the qvw daily, run the load script, save and close.

Thanks!

Tal

[swuehl]

Tal,

what about renaming / moving the qvd's at the end of your script? You can execute a command line like

execute cmd.exe /c move *.qvd ../archive;

Regards,

Stefan

[magavi_framsteg · Accepted solution]

Hi Taladmon.

There are several ways of achieving this.

One of them is:

- Create a table where you store each file you read and store it as QVD

- Check against this QVD if the file has already been read

Another way is:

- Check against a key/timestamp when you read the files

- Only load files with keys/timestamps that haven't been read before, ie. if not exists(key)

There are more ways, but these often are sufficient considering you did not supply your specific requirements.

Hope this helps.

[#4]

Thanks Magnus,

How do I do it in practice?

Currently I reload the script daily to process all the files in the directory.

Is there a way to append the data of the newest file without having the nead of reloading all the other QVDs?

Tal

[Miguel_Angel_Baeyens]

Hi Tal,

The logic for what you expect should be a double load and one store for the most recent QVD. The first loads the existing records, the second load concatenates the new data to the existing table, then you store the table (with all the data).

You can find some examples on how to get that "double step" in this thread and the macro code to load the latest file here among many others.

Hope that helps.

Miguel Angel Baeyens

BI Consultant

Comex Grupo Ibérica