Skip to main content
Announcements
Have questions about Qlik Connect? Join us live on April 10th, at 11 AM ET: SIGN UP NOW
cancel
Showing results for 
Search instead for 
Did you mean: 
Not applicable

Remove duplicate in QVD when doing incremental load

Hi everyone,

I need to do incremental load every month, however there might be some data duplicate with previous upload, may I know can I remove the duplicate rows?

** Upload ID was given to every incremental upload, therefore the row will not be exactly the same due to the Upload ID.

The following is my code:

Temp:

LOAD max(UploadID) as previousID

FROM [lib://QVD/DatasetMain.QVD](qvd);

LET vID = peek('previousID')+1;

Dataset:

LOAD

     x,

     y,

     z,

     ...

     ...

    '$(vID)' as UploadID,

     now(1) as "Date Uploaded"

FROM [lib://Raw/Data.xlsx]

(ooxml, embedded labels, table is Sheet1);

STORE [Dataset] into [lib://QVD/Dataset.QVD] (qvd);

Load * ,' ' as junk resident [Dataset];

Concatenate([Dataset])

LOAD * from [lib://QVD/DatasetMain.QVD](qvd);

Store [Dataset] into [lib://QVD/DatasetMain.QVD];

4 Replies
lakshmikandh
Specialist II
Specialist II

jonathandienst
Partner - Champion III
Partner - Champion III

To do his reliably, you either need a unique transaction ID or a last updated date in the source. It is very difficult to detect duplicates without one of those two.

Logic will get you from a to b. Imagination will take you everywhere. - A Einstein
Not applicable
Author

Hi Jonathan,

Thanks for replay but could you explain more detail?

Not applicable
Author

Hi, see this little script help for your delta load:

https://community.qlik.com/docs/DOC-17965