Load only new files

[evansabres]

I am using a QVD to load all files in a directory. The files are quite large and therefore the re-load time is long. Once a file is loaded once, there is no reason for it be loaded again as the data will not change. Is there a method to only load "new" files?

[m_woolf]

Delta Load Template

[evansabres]

My directory contains .xlsx files and is not connected to a SQL db

[m_woolf]

Those files all have FileTimes. The logic for doing an incremental load is the same.

When you first create the QVD, you load all the files and store the max Filetime in a variable.

When you do the incremental load, you load all data from the QVD. Then you concantenate load all files from the folder that have a Filetime greater than the stored max Filetime (variable).

[rwunderlich · Accepted solution]

Just add the prefix:

BUFFER (incremental)

to your LOAD statement.  That will load only new files since the last load.  The old files will come form the buffer which will be very fast.

-Rob

http://masterssummit.com

http://qlikviewcookbook.com

http://www.easyqlik.com

[evansabres]

This worked wonderfully, thank you!

[m_woolf]

Glad to help.

If any of the replies addressed your problem, please mark as Correct. If any were Helpful, please mark as Helpful.