Remove duplicate in QVD when doing incremental load

[#1]

Hi everyone,

I need to do incremental load every month, however there might be some data duplicate with previous upload, may I know can I remove the duplicate rows?

** Upload ID was given to every incremental upload, therefore the row will not be exactly the same due to the Upload ID.

The following is my code:

Temp:

LOAD max(UploadID) as previousID

FROM [lib://QVD/DatasetMain.QVD](qvd);

LET vID = peek('previousID')+1;

Dataset:

LOAD

     x,

     y,

     z,

     ...

     ...

    '$(vID)' as UploadID,

     now(1) as "Date Uploaded"

FROM [lib://Raw/Data.xlsx]

(ooxml, embedded labels, table is Sheet1);

STORE [Dataset] into [lib://QVD/Dataset.QVD] (qvd);

Load * ,' ' as junk resident [Dataset];

Concatenate([Dataset])

LOAD * from [lib://QVD/DatasetMain.QVD](qvd);

Store [Dataset] into [lib://QVD/DatasetMain.QVD];

[lakshmikandh]

Hi,

Check this one LOAD just distinct fields from one QVD and store into another qvd

[jonathandienst]

To do his reliably, you either need a unique transaction ID or a last updated date in the source. It is very difficult to detect duplicates without one of those two.

[#4]

Hi Jonathan,

Thanks for replay but could you explain more detail?

[#5]

Hi, see this little script help for your delta load:

https://community.qlik.com/docs/DOC-17965