Code to clean up duplicates

[hobanwashburne]

I have quite a few joins in my qvd creation files and have been having trouble with duplicated records. Shouldn't the following code stuck at the end of my creation (just before the store command) help with that? If so, wouldn't it be a good idea to use it in most of my qvd creation files? Does anyone else use something like this?

MasterFile:

LOAD Distinct *

Resident Temp;

DROP Table Temp;

STORE MasterFile into ....

[MK_QSL]

Temp:

Load * From.....

NoConcatenate

MasterFile:

LOAD Distinct *

Resident Temp;

DROP Table Temp;

STORE MasterFile into ....

[marcus_sommer]

With distinct you eliminated duplicated rows but you could still have duplicates in key-fields (same id but several matchings). For things like this or to have a possibilty to work on the reasons you need to identify these rows, maybe with something like this:

duplicate:

Load ID, if(count(ID) > 1, 'duplicate', 'ok') as check From x;

- Marcus