Duplicate data in incremental load

[Anonymous]

Hi All,

I am facing an issue that i am getting duplicate  set of record for particular id

I am fetching the data from qvd fetching data on daily basis

ID	OrgLvl3Flag	OrgLvl4Flag	OrgLvl5Flag	IssueFlag	OrgLvl3	OrgLvl3Old	OrgLvl4	OrgLvl4old	OrgLvl5	OrgLvl5old
123123	Change	Change	Change	Modified	XXX		YYY		ZZZ
123123	No Change	No Change	No Change	Not Modified	XXX	XXX	YYY	YYY	ZZZ	ZZZ

$(vLatestFile_Snapshot):

LOAD distinct

      [Issue ID],

     [Organisation Level 3 (Issue)] ,

     [Organisation Level 4 (Issue)] ,

     [Organisation Level 5 (Issue)] ,

  // Floor (MakeDate(2017,08,08)) as

    System_Transaction_Date

FROM

$(vLatestFileName)(qvd);

left join($(vLatestFile_Snapshot))

LOAD distinct

     [Issue ID],

     [Organisation Level 3 (Issue)] as  [Organisation Level 3 (Issue) old],

     [Organisation Level 4 (Issue)] as  [Organisation Level 4 (Issue) old],

     [Organisation Level 5 (Issue)] as  [Organisation Level 5 (Issue) old]

FROM

$(vTransformQVD)\OrionDailySnapshot\$(vPrevFile_Snapshot).QVD(qvd);

$(vOrionTableName):

Load *,

if(Orglevel3Flag='Change' or Orglevel4Flag='Change' or Orglevel5Flag='Change','Modified','Not Modified')as RiskIssueFlag;

NoConcatenate

Load distinct  *,

if([Organisation Level 3 (Issue)]= [Organisation Level 3 (Issue) old],'No Change','Change') as Orglevel3Flag,

if([Organisation Level 4 (Issue)]= [Organisation Level 4 (Issue) old],'No Change','Change') as Orglevel4Flag,

if([Organisation Level 5 (Issue)]= [Organisation Level 5 (Issue) old],'No Change','Change') as Orglevel5Flag

Resident $(vLatestFile_Snapshot);

Drop Table $(vLatestFile_Snapshot);

Concatenate($(vOrionTableName))

LOAD [Issue ID],

     [Organisation Level 3 (Issue)],

     [Organisation Level 4 (Issue)],

     [Organisation Level 5 (Issue)],

     System_Transaction_Date,

     [Organisation Level 3 (Issue) old],

     [Organisation Level 4 (Issue) old],

     [Organisation Level 5 (Issue) old],

     Orglevel3Flag,

     Orglevel4Flag,

     Orglevel5Flag,

     RiskIssueFlag

FROM

D:\WorkArea\Shweta\Org Hier data\$(vOrionTableName).qvd

(qvd);

STORE  $(vOrionTableName) into D:\WorkArea\Shweta\Org Hier data\$(vOrionTableName).qvd(qvd);

[Anil_Babu_Samineni]

May be check which table you are getting unique values by using Trace in Qlik

[arvind_patil]

Hi Shweta,

Kindly provide sample app.

Thanks,

Arvind Patil

[mayuresh_d]

Please check

"$(vPrevFile_Snapshot).QVD" this table. whether you r getting unique Issue Id or not. if that table has duplicate values then it will give you duplicate in final result as well.

[avinashelite]

Which is your incremental key ?? If feel their is a issue with your incremental logic its self ..please check the logic and data once

[Anonymous]

please find sample app

[Anonymous]

PFA  of Sample File

[Anonymous]

PFA  of Sample File

[Anonymous]

PFA  of Sample File as replied over other reply

[mayuresh_d]

please check attached app. if its same as what u want.