Incremental Load and appending files from a folder

[gerardsa]

Dear Community,

A lot of incremental posts or appending records assume that the source is residing in the same table.

I wanted to know how the incremental load would work if you have to source the new data from a different file (same file structure but different delivery date).

Assuming following Structure:

Files are deposited on irregular intervals in a folder and the file can be identified with a delivery date within the table:

Folder A:

File 1 (eg Date 1)

File 2 (eg Date 2)

File 3 (eg Date 3)

File 1 and 2 are loaded and stored in a qvd.

How would you integrate File 3 without reloading all files from folder a, store it in the qvd already in place.

After arrival of file 4, the load should consider 3 as loaded and append file 4 to the qvd and so on.

Thanks

Gerard

[kaushiknsolanki]

Hi,

My approach will be as below.

If QVD doesnt exists then
1. Load all the excels available in folder
2. Store that data in QVD using store statement.
3. Get the list of those files using file function.
4. Store those file names in QVD calles ETL.
If QVD Exists
1. Load the file name from ETL QVD.
2. Get the list of files using file function from folder.
3. Compare both the data using Exist function and get the new file names.
4. Once we have the new file names, load the data of those files and concatenate same in Existing QVD.
5. Concatenate the Filename in ETL table.

End if.

Regards,

Kaushik Solanki

[dplr-rn]

See below

Step 1 - Set up the csv file load (i.e. load only the new files

Basically for the first load (based on a flag) load the file names and data into a qvds

Subsequent loads

- get list of files names

- load old list of file names from qvd

- create another table with latest filenames only

- load data from qvd

- loop through new file names and load which gets autoconcatenated

- drop unnecessary tables (havent done this in the app to prove it works)

- save qvds

see script of attached qvw

Step 2 - implement incremental load (update and/or delete)  on the latest loads. check below  (if new excel gives updates)

https://www.analyticsvidhya.com/blog/2014/09/qlikview-incremental-load/

[Thabang231]

Hi @kaushiknsolanki  

Now in the context where I have an existing QVD 

How do i perform  these steps, and please provide example because im working with files from 3 months ago and they are daily: 

Load the file name from ETL QVD.
2. Get the list of files using file function from folder.
3. Compare both the data using Exist function and get the new file names.
4. Once we have the new file names, load the data of those files and concatenate same in Existing QVD.
5. Concatenate the Filename in ETL table.

Kindly please assist Thank you in advance.

[Thabang231]

Hi @dplr-rn 

Kindly assist with the above solution, i seem not to win on the steps. 

Thanks in advance.

[kaushiknsolanki]

@Thabang231  I dont have working example.

But did you try that, you can share the issues or the problem you faced while doing that and I will try to help you.