Incremental Load using a Loop?

Thu, 18 Apr 2013 12:54:47 GMT

Hi,

I have a requirement to take a group of monthly files, say:

data_file1_20130131

data_file2_20130131

data_file3_20130131

They are of different types and contain different data, but I merge them in the scripts and have a final output into a QVD file.

The next month I would get a new group of files in the same directory, say:

data_file1_20130228

data_file2_20130228

data_file3_20130228

I need to be able to load the new files and check if the data already exists in the QVD, I would do this by loading the QVD with a date column made up from the last 8 characters of the filename.

I want to be able to write the solution to do an incremental load, or a full reload using all the files in the directory, but one group at a time.

So, I was thinking, and would welcome some help please:

Get a list of all files in a directory and take last 8 characters as the date. This would then give me the unique number of file groups, i.e. in above example we would get 2 unique dates.
Start the load process and check if the solution qvd exists, let's say it's called FACT.qvd

If qvd does not exist then a full load can take place by looping through the group files

If qvd exists then select out of the qvd count of records for the date being processed

If no records then incremental load

Has anyone got example scripts that processes files in groups (or collections) in an iterative manner please ?

Thanks,

Shakil

Re: Incremental Load using a Loop?

Gysbert_Wassenaar — Thu, 18 Apr 2013 19:42:29 GMT

Probably easiest is to simply move the source files after they've been processed. But if you want to check the filenames instead look here.

Re: Incremental Load using a Loop?

Thu, 18 Apr 2013 21:13:38 GMT

Thank you... I will try this, but I have already tried a similar script from another site (almost identical) and QV errors saying no resident table Files not found.

Will reply back after trying.

topic Re: Incremental Load using a Loop? in QlikView

Incremental Load using a Loop?

Re: Incremental Load using a Loop?

Re: Incremental Load using a Loop?