Do not input private or sensitive data. View Qlik Privacy & Cookie Policy.
Skip to main content

Announcements
Join us in Toronto Sept 9th for Qlik's AI Reality Tour! Register Now
cancel
Showing results for 
Search instead for 
Did you mean: 
MikeGaunt
Contributor II
Contributor II

Exclude data where fields match

Hi helpful people, 

How can I exclude data from the load where fields match ? 

here is the context 

I have one table of orders and a table of orders that have shipped. 

in both tables I create a Unique id from the order number and the item line. 

In excel a vlookup of the UID from the orders table to the shipment table and then delete the matches 

So back in the Qlik world 

MikeGaunt_0-1665674586601.png

 where UID and UID2 are the same I have no need to load this data 

Open to Ideas 🙂

Labels (2)
1 Solution

Accepted Solutions
MikeGaunt
Contributor II
Contributor II
Author

Hi @Oleg_Troyansky 

 

So if I understand your Instructions 

The below should be correct. 

// this loads the data of what has shipped 
ZSHIP:

LOAD
    "Order",
    Item,
    "Order"&Item as UID2,
    "Delivery PGI date"
FROM [lib://DataFiles/ZSHIP.xlsx]
(ooxml, embedded labels, table is Format)


// this is filtering out where shipments havent take place 


where

len("Delivery PGI date">1)

;

[Format]:
LOAD
    SO,
    "Sales Document Item",
    Material,
    "Item Description",
    "Item Qty",
    "Contract Admin",
    "Ship to Name",
    "Ship to Account",
    "Ship to City",
    "Ship To",
    "First Date",
    "Item Release Status",
    "Ship Request No",
    "Delivery Note",
    [SO]& '|' &[Sales Document Item] as UID
FROM [lib://DataFiles/GCO TAB.xlsx]
(ooxml, embedded labels, table is Format)
 

 
 //   ADD FILTERS FROM gco report data to the where clause 
 //  [Item Release Status] = 'Not Released' - filters out orders that has not yet been released
 //  [First Date] <= today(1)+90 - limits the view to orders within the next three months.
 
Where

 [Item Release Status] = 'Not Released'
 
 and
 
 [First Date] <= today(1)+90
 
and

NOT Exists(UID2, [SO]& '|' &[Sales Document Item])

;

View solution in original post

9 Replies
Oleg_Troyansky
Partner Ambassador/MVP
Partner Ambassador/MVP

Hi,

this is done quite easily with the Exists() function. First load the table with one value, and then in the second load, use 

Where not exists(UID2, UID)

The first parameter is the fields that was loaded earlier and the second parameter is the field from the later load.

Cheers,

MikeGaunt
Contributor II
Contributor II
Author

Hi Oleg, 

Thanks for this I have looked up the function. 

I am having trouble positioning it correctly in the script 

below is where the UID first appears 

[Format]:
LOAD
	[SO],
	[Sales Document Item],
	[Material],
	[Item Description],
	[Item Qty],
	[Contract Admin],
	[Ship to Name],
	[Ship to Account],
	[Ship to City],
	[Ship To],
	[First Date],
	[Item Release Status],
	[Ship Request No],
	[Delivery Note],
    [SO]&[Sales Document Item] as UID,
	APPLYMAP( '__cityKey2GeoPoint', APPLYMAP( '__countryCodeAndCityName2Key', APPLYMAP( '__countryName2IsoThree', LOWER([Ship To])) & LOWER([Ship to City])), '-') AS [Format.Ship to City_GeoInfo],
	APPLYMAP( '__countryCodeIsoThree2Polygon', APPLYMAP( '__countryName2IsoThree', LOWER([Ship To])), '-') AS [Format.Ship To_GeoInfo]
 FROM [lib://BI/GCO TAB.xlsx]
 
 (ooxml, embedded labels, table is Format)

 

and in a lower section 

this is the second UID 

 

ZSHIP:

LOAD
    "Order",
    Item,
    "Order"&Item as UID,
    "Delivery PGI date"
FROM [lib://BI/ZSHIP.xlsx]
(ooxml, embedded labels, table is Format)


// this is filtering out where shipments havent take place 
where 

len("Delivery PGI date">1)

// and

// not exists (UID)



;

 

Please note the UID is to be also used for the link between the two tables

Thanks In advance

Oleg_Troyansky
Partner Ambassador/MVP
Partner Ambassador/MVP

Hi Mike,

Now I'm a bit confused... If you are using the field UID for linking the two tables, then it wouldn't be right to exclude existing UID, or nothing will ever match.

Initially I thought that your goal was to exclude data that appears in one table, from being loaded into another table, and that's what my suggestion applied to.

Please clarify your need, and we will figure it out.

MikeGaunt
Contributor II
Contributor II
Author

I will do my best to explain

I have a table of orders TABLE 1, this table consists of an Order Number (SO) and a Item (Sales Document Item) Concatenated these form the UID

 

MikeGaunt_0-1665757374744.png

MikeGaunt_1-1665757420040.png

 

Now this table contains all orders regardless of whether the orders have been shipped or not. 

 

The second table (TABLE 2) I have a list of proposed shipments, some have shipped and some have not. 

This table also includes the same UID logic

MikeGaunt_2-1665757562182.png

First step is to delete the rows from TABLE 2 where the row does not have a date 

// this loads the data of what has shipped 
ZSHIP:

LOAD
    "Order",
    Item,
    "Order"&Item as UID2,
    "Delivery PGI date"
FROM [lib://BI/ZSHIP.xlsx]
(ooxml, embedded labels, table is Format)


// this is filtering out where shipments havent take place 


where

len("Delivery PGI date">1);

 I accomplish this by evaluating the length of the field of the date as above. 

Now the second stage is to evaluate which rows in both TABLE 1 and TABLE 2 have the same UID if they do, we do not need to load them. 

No other data is required from TABLE 2 its purpose is simply to reduce the rows in TABLE 1. 

From the static report I am looking to automate with this App the instruction is as follows 

1) Delete all rows (Table 2) which do not have a date
2) Concatanate Column A and B to create UID
3) In TABLE 1, Vlookup against the UID created in column TABLE 2.
4) Delete any matches that occur between the 2 sheets.

Full Script 

[Format]:
LOAD
	[SO],
	[Sales Document Item],
	[Material],
	[Item Description],
	[Item Qty],
	[Contract Admin],
	[Ship to Name],
	[Ship to Account],
	[Ship to City],
	[Ship To],
	[First Date],
	[Item Release Status],
	[Ship Request No],
	[Delivery Note],
    [SO]&[Sales Document Item] as UID,
	APPLYMAP( '__cityKey2GeoPoint', APPLYMAP( '__countryCodeAndCityName2Key', APPLYMAP( '__countryName2IsoThree', LOWER([Ship To])) & LOWER([Ship to City])), '-') AS [Format.Ship to City_GeoInfo],
	APPLYMAP( '__countryCodeIsoThree2Polygon', APPLYMAP( '__countryName2IsoThree', LOWER([Ship To])), '-') AS [Format.Ship To_GeoInfo]
 FROM [lib://BI/GCO TAB.xlsx]
 
 (ooxml, embedded labels, table is Format)
 
 //   ADD FILTERS FROM gco report data to the where clause 
 //  [Item Release Status] = 'Not Released' - filters out orders that has not yet been released
 //  [First Date] <= today(1)+90 - limits the view to orders within the next three months.
 
Where

 [Item Release Status] = 'Not Released'
 
 and
 
 [First Date] <= today(1)+90
 
;

// this loads the data of what has shipped 
ZSHIP:

LOAD
    "Order",
    Item,
    "Order"&Item as UID2,
    "Delivery PGI date"
FROM [lib://BI/ZSHIP.xlsx]
(ooxml, embedded labels, table is Format)


// this is filtering out where shipments havent take place 


where

len("Delivery PGI date">1)



;

 

FORMAT = TABLE 1

ZSHIP = TABLE 2

Oleg_Troyansky
Partner Ambassador/MVP
Partner Ambassador/MVP

Hi Mike,

OK, I see your logic. So, after your load your Orders and Shipments, you don't need to keep the two tables linked (associated). You only load shipments to remove Orders that have been shipped already.

I'd suggest the following order of operations:

1. Load table 2 (shipments) with same the condition on the Delivery Date. Calculate UID the same way.

2. Load table 1 (orders) with the condition that the newly calculated UID does not exist in shipments:

LOAD
...
Where 
NOT Exists(UID2, [SO]&[Sales Document Item])
;
3. (optional) drop table Shipments, if you don't need it anymore.

In addition to the above, I'd recommend to add a qualifier to your concatenated field, to avoid accidental matches:


[SO]& '|' &[Sales Document Item]

This way, no similar combinations of these two numbers can be matched by accident.

Cheers,

MikeGaunt
Contributor II
Contributor II
Author

Hi @Oleg_Troyansky 

 

So if I understand your Instructions 

The below should be correct. 

// this loads the data of what has shipped 
ZSHIP:

LOAD
    "Order",
    Item,
    "Order"&Item as UID2,
    "Delivery PGI date"
FROM [lib://DataFiles/ZSHIP.xlsx]
(ooxml, embedded labels, table is Format)


// this is filtering out where shipments havent take place 


where

len("Delivery PGI date">1)

;

[Format]:
LOAD
    SO,
    "Sales Document Item",
    Material,
    "Item Description",
    "Item Qty",
    "Contract Admin",
    "Ship to Name",
    "Ship to Account",
    "Ship to City",
    "Ship To",
    "First Date",
    "Item Release Status",
    "Ship Request No",
    "Delivery Note",
    [SO]& '|' &[Sales Document Item] as UID
FROM [lib://DataFiles/GCO TAB.xlsx]
(ooxml, embedded labels, table is Format)
 

 
 //   ADD FILTERS FROM gco report data to the where clause 
 //  [Item Release Status] = 'Not Released' - filters out orders that has not yet been released
 //  [First Date] <= today(1)+90 - limits the view to orders within the next three months.
 
Where

 [Item Release Status] = 'Not Released'
 
 and
 
 [First Date] <= today(1)+90
 
and

NOT Exists(UID2, [SO]& '|' &[Sales Document Item])

;
Oleg_Troyansky
Partner Ambassador/MVP
Partner Ambassador/MVP

Hi Mike,

Almost there... Don't forget to apply the same logic with the separator in your calculation of the UID2 in the first load as well. The rest looks correct.

[SO]& '|' &[Sales Document Item] 

Cheers,

MikeGaunt
Contributor II
Contributor II
Author

Like so 

 

"Order"& '|' &Item as UID2,
Oleg_Troyansky
Partner Ambassador/MVP
Partner Ambassador/MVP

Yep, exactly right. The two key fields need to match in order for it to work.

If you are still experiencing challenges with the matching, look into the formatting of your key fields - text vs. numeric, leading zeroes, etc... Preview the data in both tables and see if you can spot any differences.