Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.
Hi everyone,
I need to do incremental load every month, however there might be some data duplicate with previous upload, may I know can I remove the duplicate rows?
** Upload ID was given to every incremental upload, therefore the row will not be exactly the same due to the Upload ID.
The following is my code:
Temp:
LOAD max(UploadID) as previousID
FROM [lib://QVD/DatasetMain.QVD](qvd);
LET vID = peek('previousID')+1;
Dataset:
LOAD
x,
y,
z,
...
...
'$(vID)' as UploadID,
now(1) as "Date Uploaded"
FROM [lib://Raw/Data.xlsx]
(ooxml, embedded labels, table is Sheet1);
STORE [Dataset] into [lib://QVD/Dataset.QVD] (qvd);
Load * ,' ' as junk resident [Dataset];
Concatenate([Dataset])
LOAD * from [lib://QVD/DatasetMain.QVD](qvd);
Store [Dataset] into [lib://QVD/DatasetMain.QVD];
Hi,
Check this one LOAD just distinct fields from one QVD and store into another qvd
To do his reliably, you either need a unique transaction ID or a last updated date in the source. It is very difficult to detect duplicates without one of those two.
Hi Jonathan,
Thanks for replay but could you explain more detail?
Hi, see this little script help for your delta load: