Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 
khaycock
Creator
Creator

Remove Duplicates in Load

I have two fields that I need to make a key on and remove any duplicates of this key. How can I do this? 

E.g:

Table1:

LOAD

Plugin,
[IP Address],
Plugin&[IP Address] AS KEY,
Field1,
Field2

FROM DATA

Labels (1)
4 Replies
Shubham_Deshmukh
Specialist
Specialist

WHat do you mean by remove duplicate?
khaycock
Creator
Creator
Author

There are duplicated versions of the KEY within the data. I want to remove that line of data.

khaycock
Creator
Creator
Author

Tenable:
LOAD Plugin,
[IP Address],
[IP Address] & '_' & Plugin AS KEY1 ,
Replace([Patch Publication Date],'UTC','') as [Patch Publication Date],
(Replace([Last Observed],'UTC','')) as [Last Observed],
(Replace([First Discovered],'UTC','')) as [First Discovered]

FROM
[data.csv]
(txt, codepage is 1252, embedded labels, delimiter is ',', msq);


Tenable_temp:
load distinct

*,
date(today(1)) - date(floor(date#([First Discovered],'MMM DD, YYYY hh:mm:ss')),'DD/MM/YYYY') as VulAgeNo,
if(date(today())-7 <= date(floor(date#([Last Observed],'MMM DD, YYYY hh:mm:ss'))), 'Scanned Recently', 'Re-Scan Needed') as ReScan
RESIDENT Tenable
WHERE NOT EXISTS (KEY1,[IP Address] & '_' & Plugin);

Drop table Tenable;

I think I followed the steps properly but this returns no results. Is it because I am forcing the key in the script?