Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.
Dear Community,
A lot of incremental posts or appending records assume that the source is residing in the same table.
I wanted to know how the incremental load would work if you have to source the new data from a different file (same file structure but different delivery date).
Assuming following Structure:
Files are deposited on irregular intervals in a folder and the file can be identified with a delivery date within the table:
Folder A:
File 1 (eg Date 1)
File 2 (eg Date 2)
File 3 (eg Date 3)
File 1 and 2 are loaded and stored in a qvd.
How would you integrate File 3 without reloading all files from folder a, store it in the qvd already in place.
After arrival of file 4, the load should consider 3 as loaded and append file 4 to the qvd and so on.
Thanks
Gerard
Hi,
My approach will be as below.
If QVD doesnt exists then
1. Load all the excels available in folder
2. Store that data in QVD using store statement.
3. Get the list of those files using file function.
4. Store those file names in QVD calles ETL.
If QVD Exists
1. Load the file name from ETL QVD.
2. Get the list of files using file function from folder.
3. Compare both the data using Exist function and get the new file names.
4. Once we have the new file names, load the data of those files and concatenate same in Existing QVD.
5. Concatenate the Filename in ETL table.
End if.
Regards,
Kaushik Solanki
See below
Step 1 - Set up the csv file load (i.e. load only the new files
Basically for the first load (based on a flag) load the file names and data into a qvds
Subsequent loads
- get list of files names
- load old list of file names from qvd
- create another table with latest filenames only
- load data from qvd
- loop through new file names and load which gets autoconcatenated
- drop unnecessary tables (havent done this in the app to prove it works)
- save qvds
see script of attached qvw
Step 2 - implement incremental load (update and/or delete) on the latest loads. check below (if new excel gives updates)
https://www.analyticsvidhya.com/blog/2014/09/qlikview-incremental-load/
Now in the context where I have an existing QVD
How do i perform these steps, and please provide example because im working with files from 3 months ago and they are daily:
Load the file name from ETL QVD.
2. Get the list of files using file function from folder.
3. Compare both the data using Exist function and get the new file names.
4. Once we have the new file names, load the data of those files and concatenate same in Existing QVD.
5. Concatenate the Filename in ETL table.
Kindly please assist Thank you in advance.
Hi @dplr-rn
Kindly assist with the above solution, i seem not to win on the steps.
Thanks in advance.
@Thabang231 I dont have working example.
But did you try that, you can share the issues or the problem you faced while doing that and I will try to help you.