topic Store new rows into QVD without loading QVD? (Qlik Sense) in App Development

Store new rows into QVD without loading QVD? (Qlik Sense)

hjm35055 — Sat, 16 Nov 2024 18:10:16 GMT

Hi,

I am wondering if, in Qlik Sense, there is a way to store new rows into a qvd without needing to load the qvd.

For example:

I am doing an incremental load off of a date and there are 100 new rows of data.
The qvd that I need to store those 100 new rows into contains over 60 million rows

Is there a way to store those 100 new rows into the qvd with 60 million rows without having to load the 60 million rows?

Current Script

Incremental_Table:
NoConcatenate
Load
*
Resident Table1
Where IsNull(deletedDateTime) // Only loading rows that do not have a deleted timestamp
;

Drop Table table1;

// Qvd with over 60 million rows
Concatenate (Incremental_Table)
LOAD
*
FROM [lib://QlikData/Fact.qvd] (qvd);

Store Incremental_Table into [lib://QlikData/Fact.qvd] (qvd);

Drop TableIncremental_Table;

Thank you!

Re: Store new rows into QVD without loading QVD? (Qlik Sense)

TiagoCardoso — Mon, 31 Aug 2020 18:49:48 GMT

Olá.

Sorry for my english.

Maybe helpful.

https://www.analyticsvidhya.com/blog/2014/09/qlikview-incremental-load/

this article is in qlik view, but you can do the same for qlik sense without any problems.

Re: Store new rows into QVD without loading QVD? (Qlik Sense)

hjm35055 — Mon, 31 Aug 2020 22:53:25 GMT

Thanks for the article. I am currently doing an incremental load.

In the example below, lines 34-41, they are concatenating the existing qvd to the new rows. My issue is, it is a large qvd (over 60 million rows) and I was wondering if there was a way to store the new rows into the qvd without having to load the existing qvd into the app. I think the answer is no, but figured I would ask.

Re: Store new rows into QVD without loading QVD? (Qlik Sense)

marcus_sommer — Tue, 01 Sep 2020 08:09:26 GMT

It's not possible just to append any records to a qvd without loading them because Qlik doesn't use a linear row-storing else it's a column-based storing-format which only stored the distinct values of each fields and creates a bit-stuffed pointer to link the values with the records. This logic/structure needs to re-created by each change.

Usually this isn't a problem because an optimized load from a qvd is very fast even by large datsets because no extra/new processing is needed to load the data else they are just transferred from the storage/network into the RAM - means it depends completely from their performance how long it takes.

Beside this I suggest to reverse the order within your incremental approach and loading at first the larger historical qvd-data and then adding the new records.

- Marcus