topic Re: Daily CSV incremental load in QlikView

Daily CSV incremental load

userid128223 — Wed, 10 Aug 2016 21:45:48 GMT

Hi Everyone

I have following scenerio. I have source folder where data arrives on daily bases and i need to create 1 qvd for all the data without duplicaiton. I also need to read this folder daily and load any new files that arrive.

C:\SourceFolder\

Day1:

Tax20150801.csv

Tax20150802.csv

Tax20150803.csv

Tax20150804.csv

Load * > accumulated_Tax.csv

Day2:

Tax20150801.csv

Tax20150802.csv

Tax20150803.csv

Tax20150804.csv

Tax20150805.csv

Tax20150806.csv

Idea is on day 2 only 2 new file will be appended to accumulated_Tax.csv and others will be ignored. Also make sure that if this does not run daily, it will take any new files that was not processed previously. I would also like to keep a log of files that were processed.

Thanks

Re: Daily CSV incremental load

swuehl — Wed, 10 Aug 2016 22:25:49 GMT

Store the filename of each file into the accumulated_Tax.qvd (I would store the resulting table into a QVD), using e.g FileBaseName() to create the new field.

Then your script basically looks like

a) LOAD your historic data, the accumulated_Tax.qvd

b) Using a FOR EACH vFile IN FILELIST('*.csv') ... NEXT vFile, loop over all csv file in your directory

- Inside the loop, use an IF condition based on the filename and a lookup of the stored filename to check if it already exists in the historic table. If it doesn't exist, append the csv table to your historic table.

Re: Daily CSV incremental load

userid128223 — Thu, 11 Aug 2016 00:44:01 GMT

Hi Stefan

Can you please point me to a script that does this if already exist.

Thanks

Re: Daily CSV incremental load

michael_klix — Thu, 11 Aug 2016 10:16:17 GMT

‌Hi Stefan,

usually, my scripts move all read files to an archive directory and add a field "datasource" containing the filename just for referrence in case the dataset is delivered again in another file. Is that possible in your environment?

keep qliking

michael

Re: Daily CSV incremental load

userid128223 — Thu, 11 Aug 2016 13:11:10 GMT

I am aware of that method but I think there are chances of error calling windows command in qlikview. I think if you keep a file database as to what got loaded and what requires loading is good way to avoid errors.

Re: Daily CSV incremental load

userid128223 — Mon, 22 Aug 2016 22:08:46 GMT

Any examples on this