Qlik Community

Ask a Question

Community Corner

Discussion board for questions about the Qlik Community, its features, sharing information, general discussions and even some fun. This is for both new and longtime community members. Everyone is welcome!

Talk to Experts Tuesday, January 26th at 10AM EST: Qlik Sense. REGISTER NOW
Showing results for 
Search instead for 
Did you mean: 

Incremental Load and appending files from a folder

Dear Community,

A lot of incremental posts or appending records assume that the source is residing in the same table.

I wanted to know how the incremental load would work if you have to source the new data from a different file (same file structure but different delivery date).

Assuming following Structure:

Files are deposited on irregular intervals in a folder and the file can be identified with a delivery date within the table:

Folder A:

File 1 (eg Date 1)

File 2 (eg Date 2)

File 3 (eg Date 3)


File 1 and 2 are loaded and stored in a qvd.

How would you integrate File 3 without reloading all files from folder a, store it in the qvd already in place.

After arrival of file 4, the load should consider 3 as loaded and append file 4 to the qvd and so on.



2 Replies
MVP & Luminary
MVP & Luminary


My approach will be as below.

If QVD doesnt exists then
1. Load all the excels available in folder
2. Store that data in QVD using store statement.
3. Get the list of those files using file function.
4. Store those file names in QVD calles ETL.
If QVD Exists
1. Load the file name from ETL QVD.
2. Get the list of files using file function from folder.
3. Compare both the data using Exist function and get the new file names.
4. Once we have the new file names, load the data of those files and concatenate same in Existing QVD.
5. Concatenate the Filename in ETL table.

End if.



Kaushik Solanki

If reply is satisfactory, please "Like" the post.
If reply is solution, please mark as "Correct".

See below

Step 1 - Set up the csv file load (i.e. load only the new files

Basically for the first load (based on a flag) load the file names and data into a qvds

Subsequent loads

- get list of files names

- load old list of file names from qvd

- create another table with latest filenames only

- load data from qvd

- loop through new file names and load which gets autoconcatenated

- drop unnecessary tables (havent done this in the app to prove it works)

- save qvds

see script of attached qvw


Step 2 - implement incremental load (update and/or delete)  on the latest loads. check below  (if new excel gives updates)