Comparing filelist to table on load

Tue, 11 Sep 2012 13:17:12 GMT

I am (incrementally) loading data from CSV-files, located in different folders.

Each day, more (hundreds of) csv-files will be added to these folders, and they will not (cannot) be (re)moved after load.

To loop through the folders I am using the "Sub DoDir(Root)" from the user's guide

And the using:

// CODE

For Each File In FileList ('$(FileRoot)')

LOAD...

variable_1,

variable_2,

etc...

Form [$(File)](txt)

Next File

// CODE END

to load data from the csv-files therein.

NOW - I also have a qvd-file with a table with the names of the files previously processed.

Fileldname is Previous_filenames

What I need to do is to compare the filename of each csv-file to the list of previously loaded files, and only load the ones not previously loaded.

Something like:

If not exists(Previous_filenames, Filename('$(File)') Then...

Any ideas?

Re: Comparing filelist to table on load

whiteline — Tue, 11 Sep 2012 13:37:05 GMT

You've already done half of it, assigned a name.

Now create a field Previous_filenames, just add it to your LOADs:

LOAD...

variable_1,

variable_2,

SubField('$(File)', '\', Substringcount('$(File)')+1) as Previous_filenames

Form [$(File)](txt)