Qlik Community

New to Qlik Community

Discussion board for questions on how to use Qlik Community and its features.

Highlighted
gerardsa
New Contributor

Incremental Load and appending files from a folder

Dear Community,

A lot of incremental posts or appending records assume that the source is residing in the same table.

I wanted to know how the incremental load would work if you have to source the new data from a different file (same file structure but different delivery date).

Assuming following Structure:

Files are deposited on irregular intervals in a folder and the file can be identified with a delivery date within the table:

Folder A:

File 1 (eg Date 1)

File 2 (eg Date 2)

File 3 (eg Date 3)

 

File 1 and 2 are loaded and stored in a qvd.

How would you integrate File 3 without reloading all files from folder a, store it in the qvd already in place.

After arrival of file 4, the load should consider 3 as loaded and append file 4 to the qvd and so on.

Thanks

Gerard

2 Replies

Re: Incremental Load and appending files from a folder

Hi,

My approach will be as below.

If QVD doesnt exists then
1. Load all the excels available in folder
2. Store that data in QVD using store statement.
3. Get the list of those files using file function.
4. Store those file names in QVD calles ETL.
If QVD Exists
1. Load the file name from ETL QVD.
2. Get the list of files using file function from folder.
3. Compare both the data using Exist function and get the new file names.
4. Once we have the new file names, load the data of those files and concatenate same in Existing QVD.
5. Concatenate the Filename in ETL table.

End if.

 

Regards,

Kaushik Solanki

dilipranjith
Valued Contributor II

Re: Incremental Load and appending files from a folder

See below

Step 1 - Set up the csv file load (i.e. load only the new files

Basically for the first load (based on a flag) load the file names and data into a qvds

Subsequent loads

- get list of files names

- load old list of file names from qvd

- create another table with latest filenames only

- load data from qvd

- loop through new file names and load which gets autoconcatenated

- drop unnecessary tables (havent done this in the app to prove it works)

- save qvds

see script of attached qvw

 

Step 2 - implement incremental load (update and/or delete)  on the latest loads. check below  (if new excel gives updates)

https://www.analyticsvidhya.com/blog/2014/09/qlikview-incremental-load/

 

Community Browser