topic Incremental load from multiple QVD files in QlikView

Incremental load from multiple QVD files

Mon, 22 Aug 2011 14:47:19 GMT

Hi all,

Our QV module reads data from multiple QVD files stores in a specific directory.

Each day new file is being added to the same directory.

How can we make the load of the qvw to take only the newly added files (since last update)?

Currently, we open the qvw daily, run the load script, save and close.

Thanks!

Tal

Re: Incremental load from multiple QVD files

swuehl — Mon, 22 Aug 2011 14:56:01 GMT

Tal,

what about renaming / moving the qvd's at the end of your script? You can execute a command line like

execute cmd.exe /c move *.qvd ../archive;

Regards,

Stefan

Re: Incremental load from multiple QVD files

magavi_framsteg — Mon, 22 Aug 2011 14:59:39 GMT

Hi Taladmon.

There are several ways of achieving this.

One of them is:

- Create a table where you store each file you read and store it as QVD

- Check against this QVD if the file has already been read

Another way is:

- Check against a key/timestamp when you read the files

- Only load files with keys/timestamps that haven't been read before, ie. if not exists(key)

There are more ways, but these often are sufficient considering you did not supply your specific requirements.

Hope this helps.

Incremental load from multiple QVD files

Tue, 23 Aug 2011 11:54:00 GMT

Thanks Magnus,

How do I do it in practice?

Currently I reload the script daily to process all the files in the directory.

Is there a way to append the data of the newest file without having the nead of reloading all the other QVDs?

Tal

Incremental load from multiple QVD files

Miguel_Angel_Baeyens — Tue, 23 Aug 2011 12:14:16 GMT

Hi Tal,

The logic for what you expect should be a double load and one store for the most recent QVD. The first loads the existing records, the second load concatenates the new data to the existing table, then you store the table (with all the data).

You can find some examples on how to get that "double step" in this thread and the macro code to load the latest file here among many others.

Hope that helps.

Miguel Angel Baeyens

BI Consultant

Comex Grupo Ibérica