I am unsuccessful in combining Not Exists and qualify/unqualify.
Goal: load a dataset and identify lines that have one field (or combinaton of fields) in common with previously loaded lines; in that case either not load the line (what I try to do here) or as an alternative load the line with a special flag telling it is a "duplicate".
If I do this :
table1:
LOAD
a,
b
FROM
file.txt
(...)
where
not exists(a);
It works fine, I have no duplicate lines for field a.
If I do this:
qualify*;
table1:
LOAD
a,
b
FROM
file.txt
(...)
where
not exists(a);
unqualify;
It doesn't work, I get an empty table.
If I do this :
qualify*;
table1:
LOAD
a,
b
FROM
file.txt
(...)
where
not exists(table1.a);
unqualify;
It doesn't work, I get an error message from the load telling the field table1.a cannot be found.
