topic Re: Qlik Sense Data Load - Remove duplicate records in App Development

Qlik Sense Data Load - Remove duplicate records

jerryr125 — Fri, 26 Mar 2021 17:01:09 GMT

Hi - I have the following in my data load editor:

RPSTATUSDATA03:
Noconcatenate
Load *
RESIDENT RPQUISITIONSTATUSDATA02;

I need to remove any records in which there are duplicates.

The field which determines this is the RPNUMMAT field.

If the RPNUMMAT is the same then I do not need the record in RPSTATUSDATA03.

Any thoughts on how I can achieve this process ?

Thanks so much - Jerry

Re: Qlik Sense Data Load - Remove duplicate records

rubenmarin — Sat, 27 Mar 2021 07:48:55 GMT

Hi, you can use Exists() to check already loaded values:

RPSTATUSDATA03: Noconcatenate Load *, RPNUMMAT as chkRPNUMMAT RESIDENT RPQUISITIONSTATUSDATA02 Where not exists('chkRPNUMMAT ',RPNUMMAT); DROP Field chkRPNUMMAT;

As all field have been loaded in RPQUISITIONSTATUSDATA02, you have to cahnge field name to check those values loaded in RPSTATUSDATA03.

Re: Qlik Sense Data Load - Remove duplicate records

jerryr125 — Tue, 30 Mar 2021 11:53:59 GMT

Hi- I finally had the opportunity to try the above - thank you very much - works perfectly !!

- Jerry

Re: Qlik Sense Data Load - Remove duplicate records

Dalton_Ruer — Tue, 30 Mar 2021 13:03:07 GMT

Have you tried LOAD DISTINCT?
Refer to the help documentation here: https://help.qlik.com/en-US/sense/February2021/Subsystems/Hub/Content/Sense_Hub/Scripting/ScriptRegularStatements/Load.htm