topic appending data in QlikView

appending data

Tue, 17 Apr 2012 22:33:10 GMT

Hi,

I have just a question, how can I append some data to a QVD ?

I have read the documentation about incremental load but I have understood what I'm going to explain here below.

Ok, I can extract just new data from a database server (suppose we ha ve no update or delete) and this is powerful.

I can then concatenate this data with the consolidated data one stored in a QVD, and here is the issue.

To performe a concatenation I have to read all the data from a qvd into the memory, and this is NOT what I want (because a performance issue and waste time).

I just want to write in APPEND some data into a qvd without have to load it and overwrite it.

I haven't understood if it's possible or not (or if I have always to store the new qvd with the concatenation of old and new data together).

Of course, if we work with bigdata, this can be a big issue and can maybe sometimes occurs in memory overflow.

Thanks a lot

appending data

adhudson — Wed, 18 Apr 2012 03:16:16 GMT

Hi,

If I misunderstood your question, please correct.

Here is the answer:

Whenever the qlikview file gets reloaded, that means it clears the preloaded data from memory and reloads the new data from the file again.

This is what generally happening in qlikview reload.

But in your case, You just want load the new data which you have downloaded from the data source recently and without loading the previous data i.e., the exact previously loaded table, I don't think it is possible.

Because, to concatenate the new data with the table that should exist in the memory before the concatenation process. right?

I think it is must to load the previous table for concatenation. Only Loading the recent data will throw an error logically. Lets say it would be a table not found error.

Regards

Andrew Hudson

appending data

Wed, 18 Apr 2012 22:18:32 GMT

Hi,

I just want to store the new data extracted from the datasource in an appending mode to an existing qvd.

I think this is not possible because in every documentation file I have read it is always described that you have to concatenate two dataset (the historical qvd and the new data extracted) and store in the qvd (overwriting the old one)...but concatenate means that I have to load data into the memory.

Can you confirm this ?

If so, I'll just stop to search

Thanks a lot

appending data

Wed, 18 Apr 2012 22:50:19 GMT

that is correct, you can't append to a QVD without loading it first into memory to concatenate to it.

I understand you can see it as a time and performance issue, but unless there's an operation performed on the old data, it should be read in optimized mode and shouldn't be an issue.

appending data

adhudson — Thu, 19 Apr 2012 00:13:21 GMT

Ya... I confirm that you need to load the data before concatenation.

Regards

Andrew Hudson

appending data

rwunderlich — Thu, 19 Apr 2012 01:58:48 GMT

As others have confirmed, there is no Append functionality for QVDs. As an alternative to re-reading and merging into a single QVD, you can store each set of updates into a seperate QVD file, qualifed for example with a datestamp. For example:

Transactions_20120417.qvd

And then your QVD consuming script can load all QVDs with a wildcard.

LOAD * FROM Transactions_*.qvd (qvd).

-Rob

http://robwunderlich