topic Re: Generic Load is creating duplicate records in App Development

Generic Load is creating duplicate records, how to overcome this in Qlik Sense?

Prabhu — Thu, 03 Nov 2022 13:54:00 GMT

Hi All,

I am trying to create a generic load with the below-given script. But the output table(Unique) is having duplicate records. Actually, it should have unique records.

SchemaData: LOAD distinct "Case ID", taskName, taskStatus, SchemaAge, SchemaStartTimestamp, SchemaAgeAtTimestamp FROM [lib://BOB/qvd/SchemaData.qvd] (qvd); Unique: LOAD Distinct "Case ID" RESIDENT SchemaData; task: GENERIC LOAD "Case ID", taskName &':Task Status', taskStatus RESIDENT SchemaData; drop table SchemaData; FOR i = NoOfTables()-1 to 0 STEP -1 LET vTable=TableName($(i)); IF WildMatch('$(vTable)', 'task*') THEN LEFT JOIN (Unique) LOAD * RESIDENT [$(vTable)]; DROP TABLE [$(vTable)]; ENDIF NEXT i

@Saravanan_Desingh @vamsee

Re: Generic Load is creating duplicate records

canerkan — Thu, 17 Feb 2022 11:54:56 GMT

Hi Prabhu,

maybe try this change in your Script:

SchemaData:
LOAD distinct
"Case ID",
taskName,
taskStatus,
SchemaAge,
SchemaStartTimestamp,
SchemaAgeAtTimestamp
FROM [lib://BOB/qvd/SchemaData.qvd]
(qvd);

Unique:
LOAD Distinct
"Case ID"
RESIDENT SchemaData;

task:
GENERIC LOAD "Case ID", taskName &':Task Status', taskStatus
RESIDENT SchemaData;

drop table SchemaData;

FOR i = NoOfTables()-1 to 0 STEP -1
LET vTable=TableName($(i));
if SubField(vTable,'.',1) = 'task' THEN
LEFT JOIN (Unique) LOAD * RESIDENT [$(vTable)];
DROP TABLE [$(vTable)];
ENDIF
NEXT i

Maybe there are Tables with Names that also contain the word 'task*' and are therfore joined without being part of the generic load.

Let me know if it helped.

Regards,

Can.

Re: Generic Load is creating duplicate records

Prabhu — Thu, 17 Feb 2022 13:07:23 GMT

Hi canerkan,
Thanks for the update. But it's not working. I am still getting duplicate
records.

Re: Generic Load is creating duplicate records

canerkan — Thu, 03 Nov 2022 06:06:55 GMT

Please check if there are any duplicates in the Table you are using for your generic load. It could be a Data Quality Issue.

Re: Generic Load is creating duplicate records

Prabhu — Fri, 18 Feb 2022 06:26:46 GMT

Yes Canerkan, my data has duplicates that is causing the problem. There is no problem with generic load.