I am trying to do some incremental download using QVD and adding more files and store it to a n updated QVD file.
The Data contains time stamp. So far I have downloaded some QVD files. Now, I'd like to add some csv files with criteria where the latest time stamp is greater than the latest time stamp in QVD file. What I did was
Incremental_CDR:
LOAD Distinct
[Time Stamp], B,C,D,E,F,
FROM [..\QVD\Filename.qvd] (qvd);
//Last_Record:
LOAD Max (timestamp([Time Stamp])) as Latest_Time_Stamp
Resident Incremental_CDR;
LET $(vLast_Record)= Peek('[Latest_timestamp]', 0, 'Last_Record)');
trace >>> $(vLast_Record);
//Load new Records
concatenate
LOAD Distinct
FROM
$(vSourceData)Filename_*.csv
(txt, codepage is 1252, embedded labels, delimiter is ',', msq)
Where Timestamp([Time Stamp]) > $(vLast_Record);
but it failing with error as follows:
Field not found - <>>
concatenate
LOAD Distinct ...... and rest of the message
When I changed the $(vLast_Record) to a date(e.g. '09/05/2016') it works, but I want to to make it more precise and need to rely on the latest time stamp in last record so that I can append the qvd data and next time do another incremental load will simply use the latest time stamp in the latest qvd file or last record in another source.
It is very helpful and now I'm getting closer to the final step, but not there yet. The error is gone after fixing the statement and trace shows value of the last record in initial run, but when running with the new record no new records are added or concatenated. It only load the qvd file that has only the initial record although the condition to include the new records is met.
// Load new records
Concatenate
Load distinct
Field names
From source (*.csv)
Where > 'vLast_Record';
Am I missing something?
Additionally, I'd also like to update the latest time stamp with the latest record in the latest loaded file.
