Load files with date in filename greater than a variable

[#1]

Hi

Probably an easy one for you.

Every day I receive a data file named: et_open_DDMMYYYY.txt where DDMMYYYY is the date the file is sent to me. I have put all the data in those files into a QVD and I load the new records from time to time (not daily) using a simple incremental load. This works fine but I know that there are only new records in files where the date in the file name is greater than the last reload date.

So every time I reload, the script loads the 'historical' QVD plus the new records from my daily text files (LOAD... FROM et_open_*.txt), which means that the script goes and searches each file for new records, so loading time is longer than necessary.

I have made the last reload timestamp into the value of a variable called vScriptReload:

LET vScriptReload=RELOADTIME()

No idea how to tell it to look in the filename and load only those where the date is greater than the variable.

Any ideas?

Lisa

[#2]

Hello Lisa, you can use a where clause combined with the function filename() and perform some string operations to get the actual date from the file name. Once you accomplish this you would want to use the function makedate to create a date object corresponding to the string or strings you extracted from the filename.

Hope this helps you

Regards

[rwunderlich]

Lisa,

You can use a FOR EACH filelist() loop to conditionaly process the files. But before going down that road, have you tried adding a BUFFER statement to your load? That's what I generally do in this situation. BUFFER makes it run so fast it is usually not worth the trouble to code up the loop logic.

BUFFER LOAD... FROM et_open_*.txt;

-Rob