1 Reply Latest reply: Dec 31, 2013 3:46 PM by Srikanth P RSS

    Clean up script to remove dupicates

      All,

       

      would you have reference to a clean up script to remove 'quasi-duplicates' in a straight table ?

       

      examples include

       

      Ex1.

      Field1   Field2   Field3  

      Val1     Val2     name

      Val1     Val2     name.

      ...

      merged to

       

      Field1   Field2   Field3  

      Val1     Val2     name

      ...

       

      Ex2.

      Field1   Field2   Field3  

      Val1     Val1     Val1

      Val1     N/A      N/A

      ...

      merged to

       

      Field1   Field2   Field3  

      Val1     Val1     Val1

      ...

       

      QU1. from BI point of view: what clean up makes sense in sales database when it comes to clean account entries (this is a general and open question) ?

       

      QU2. would you have script to perform the two above operations in example, or other simple clean up operations ? also simple script for duplicates removal ?

       

      Thanks!

        • Re: Clean up script to remove dupicates
          Srikanth P

          Please use DISTINCT key word after LOAD statement to avoid the duplicate rows.

           

          TEMP:

          LOAD Field1, Field2, Field3 FROM SOURE1;

           

          LOAD Field1, Field2, Field3 FROM SOURE3;

           

          LOAD Field1, Field2, Field3 FROM SOURE3;

           

          FINAL:

          Noconcatenate

          LOAD DISTINCT * Resident TEMP ;

           

          DROP Table TEMP;