Clean up script to remove dupicates

[#1]

All,

would you have reference to a clean up script to remove 'quasi-duplicates' in a straight table ?

examples include

Ex1.

Field1   Field2   Field3  

Val1     Val2     name

Val1     Val2     name.

...

merged to

Field1   Field2   Field3  

Val1     Val2     name

...

Ex2.

Field1   Field2   Field3  

Val1     Val1     Val1

Val1     N/A      N/A

...

merged to

Field1   Field2   Field3  

Val1     Val1     Val1

...

QU1. from BI point of view: what clean up makes sense in sales database when it comes to clean account entries (this is a general and open question) ?

QU2. would you have script to perform the two above operations in example, or other simple clean up operations ? also simple script for duplicates removal ?

Thanks!

[#2]

Please use DISTINCT key word after LOAD statement to avoid the duplicate rows.

TEMP:

LOAD Field1, Field2, Field3 FROM SOURE1;

LOAD Field1, Field2, Field3 FROM SOURE3;

LOAD Field1, Field2, Field3 FROM SOURE3;

FINAL:

Noconcatenate

LOAD DISTINCT * Resident TEMP ;

DROP Table TEMP;