4 Replies Latest reply: Jan 19, 2017 4:12 AM by Sabine Boehme RSS

    Remove duplicate in QVD when doing incremental load

    lingling Guo

      Hi everyone,

       

      I need to do incremental load every month, however there might be some data duplicate with previous upload, may I know can I remove the duplicate rows?

      ** Upload ID was given to every incremental upload, therefore the row will not be exactly the same due to the Upload ID.

       

       

      The following is my code:

      Temp:

      LOAD max(UploadID) as previousID

      FROM [lib://QVD/DatasetMain.QVD](qvd);

      LET vID = peek('previousID')+1;

       

      Dataset:

      LOAD

           x,

           y,

           z,

           ...

           ...

          '$(vID)' as UploadID,

           now(1) as "Date Uploaded"

      FROM [lib://Raw/Data.xlsx]

      (ooxml, embedded labels, table is Sheet1);

       

       

      STORE [Dataset] into [lib://QVD/Dataset.QVD] (qvd);

      Load * ,' ' as junk resident [Dataset];

      Concatenate([Dataset])

      LOAD * from [lib://QVD/DatasetMain.QVD](qvd);

      Store [Dataset] into [lib://QVD/DatasetMain.QVD];