Skip to main content
Announcements
Have questions about Qlik Connect? Join us live on April 10th, at 11 AM ET: SIGN UP NOW
cancel
Showing results for 
Search instead for 
Did you mean: 
anushahegde
Contributor III
Contributor III

Disadvantages of buffer load

Hi all,

Could you help me to list out the disdavantages of buffer load?

Thanks in advance!!

1 Solution

Accepted Solutions
Peter_Cammaert
Partner - Champion III
Partner - Champion III

No. The only one that works reliably with databases is the STALE AFTER... BUFFER LOAD. That means that the Buffered QVD will serve as a cache until it expires. At that time, the entire table will be refreshed from the source.


By definition, a LOAD/SELECT statement has no idea about how your table is organized. And the DBMS won't be able to tell your SELECT statement how to perform an incremental load behind the scenes. You have to explicitly tell the SELECT how your table is organised and how to select new/changed records and only read those. At that point, you're creating an Incremental load statement yourself. Which is your only option in many if not all cases to get a reliably working Incremental Load.

View solution in original post

9 Replies
Anil_Babu_Samineni

I haven't seen any disadvantages AON

Please add me Anil_Babu_Samineni to interact faster when reply back. Speak low think High.

Before develop something, think If placed (The Right information | To the right people | At the Right time | In the Right place | With the Right context)
madhumitha
Creator
Creator

Hello Anusha,

please check the below video...

Episode 29 Buffering (caching) a Load Statement - YouTube

ramasaisaksoft

Buffer

QVD files can be created and maintained automatically via the buffer prefix. This prefix can be used on most Load and Select (SQL) statements in script. It indicates that QVD Files are used to cache/buffer the result of the statement.

Numerous limitations exist, most notable is that there must be either a file load or a select statement at the core of any complex statement.

The name of the QVD file is a calculated name (a 160-bit hexadecimal hash of the entire following load or select statement and other discriminating info) and is typically stored in the Application Data folder or another folder specified under User Preferences: Locations. This means that the QVD buffer will be rendered invalid by any change in the following load or select statement.

QVD buffers will normally be removed when no longer referenced anywhere throughout a complete script execution in the document that created it or when the document that created it no longer exists. This housekeeping can be turned off by checking the options Keep Unreferenced QVD Buffers in Document Properties: General (first requirement) and Keep Orphaned QVD Buffers in User Preferences (second requirement) property pages respectively, though the procedure cannot be generally recommended.

The syntax is:

buffer[ (option [ , option] ) ] ( load statement | select statement )

where:

option ::= incremental | expiry

expiry::= stale [after]amount[ (days | hours)]

amount is a number specifying the time period. Decimals may be used. The unit is assumed to be days if omitted.

The incremental option enables the ability to read only part of an underlying file. Previous size of the file is stored in the XML header in the QVD file. This is particularly useful with log files. All records loaded at a previous occasion are read from the QVD file whereas the following new records are read from the original source and finally an updated QVD-file is created. Note that the incremental option can only be used with load statements and text files and that incremental load cannot be used where old data is changed or deleted!

The stale after option is typically used with DB sources where there is no simple timestamps on the original data. Instead one specifies how old the QVD snapshot can be to be used. A stale after clause simply states a time period from the creation time of the QVD buffer after which it will no longer be considered valid. Before that time the QVD buffer will be used as source for data and after that the original data source will be used. The QVD buffer file will then automatically be updated and a new period starts.

If no option is used, the QVD buffer created by the first execution of the script will be used indefinitely.

Examples:

buffer select * from My Table;
buffer (stale after 7 days) select * from My Table;
buffer (incremental) load * from My Log.log;

-->so as i know  ,QVD 's are stored in default location only not happy to developer(Extract qvd's need to save one folder,Transform qvd's

need to store some other location..etc.)

-->that's way Most of the time we will use Store command only while creating the qvd's

HirisH_V7
Master
Master

I think one main thing,

The buffer stored qvd's have a temporary character for a certain app itslef and normal stored qvd's are a method to store data in a long term.For reusability.

-Hirish

HirisH
“Aspire to Inspire before we Expire!”
anushahegde
Contributor III
Contributor III
Author

Thank you.

After the buffered qvds are generated when we reload the data again in QV, will the buffered qvds gets refreshed with the new data in the database?

anushahegde
Contributor III
Contributor III
Author

Thank you!!

anushahegde
Contributor III
Contributor III
Author

Thank you !!

Peter_Cammaert
Partner - Champion III
Partner - Champion III

No. The only one that works reliably with databases is the STALE AFTER... BUFFER LOAD. That means that the Buffered QVD will serve as a cache until it expires. At that time, the entire table will be refreshed from the source.


By definition, a LOAD/SELECT statement has no idea about how your table is organized. And the DBMS won't be able to tell your SELECT statement how to perform an incremental load behind the scenes. You have to explicitly tell the SELECT how your table is organised and how to select new/changed records and only read those. At that point, you're creating an Incremental load statement yourself. Which is your only option in many if not all cases to get a reliably working Incremental Load.

anushahegde
Contributor III
Contributor III
Author

This is what I was looking for. Thank you!!