Daily CSV incremental load

[userid128223]

Hi Everyone

I have following scenerio. I have source folder where data arrives on daily bases and i need to create 1 qvd for all the data without duplicaiton. I also need to read this folder daily and load any new files that arrive.

C:\SourceFolder\

Day1:

Tax20150801.csv

Tax20150802.csv

Tax20150803.csv

Tax20150804.csv

Load * > accumulated_Tax.csv

Day2:

Tax20150801.csv

Tax20150802.csv

Tax20150803.csv

Tax20150804.csv

Tax20150805.csv

Tax20150806.csv

Idea is on day 2 only 2 new file will be appended to accumulated_Tax.csv and others will be ignored. Also make sure that if this does not run daily, it will take any new files that was not processed previously. I would also like to keep a log of files that were processed.

Thanks

[swuehl]

Store the filename of each file into the accumulated_Tax.qvd (I would store the resulting table into a QVD), using e.g FileBaseName() to create the new field.

Then your script basically looks like

a) LOAD your historic data, the accumulated_Tax.qvd

b) Using a FOR EACH vFile IN FILELIST('*.csv') ... NEXT vFile, loop over all csv file in your directory

     - Inside the loop, use an IF condition based on the filename and a lookup of the stored filename to check if it already exists in the historic table. If it doesn't exist, append the csv table to your historic table.

[userid128223]

Hi Stefan

Can you please point me to a script that does this if already exist.

Thanks

[michael_klix]

‌Hi Stefan,

usually, my scripts move all read files to an archive directory and add a field "datasource" containing the filename just for referrence in case the dataset is delivered again in another file. Is that possible in your environment?

keep qliking

michael

[userid128223]

I am aware of that method but I think there are chances of error calling windows command in qlikview. I think if you keep a file database as to what got loaded and what requires loading is good way to avoid errors.

[userid128223]

Any  examples on this