Duplicates when merging tables via resident load

[Anonymous]

I am trying to merge/concatenate two tables into one table so I can apply logic on fields from both of the tables ie

if Region like 'Florida*' and Sales_Person like 'Andy*, 'Marketing','HR' as New_Field

This is my code so far but what happens is I seem to get a huge number of duplicates and it generally looks off. Please advise ...

Table_A:

Load

Sales_ID,

Region,

Unit

From QVD1

Table_B:

Load

Sales_ID

Sales_Person,

Country

From QVD2

All_Profiles:

NoConcatenate

Load *

Resident Table_A ;

Concatenate

LOAD *,

if Region like 'Florida*' and Sales_Person like 'Andy*, 'Marketing','HR' as New_Field

Resident Table_B;

DROP TABLES Table_A ,Table_B;

[sunny_talwar]

Which field are you seeing duplicate in? or the whole row is duplicating?

[Anonymous]

The whole row. Do I need to drop the Sales_ID field perhaps ?

[sunny_talwar]

May be you can try this then

All_Profiles:

NoConcatenate

Load DISTINCT *

Resident Table_A ;

Concatenate

LOAD DISTINCT *,

    if Region like 'Florida*' and Sales_Person like 'Andy*, 'Marketing','HR' as New_Field

Resident Table_B;

DROP TABLES Table_A ,Table_B;

[Anonymous]

I have already tried that to no avail. I think it's because the new table that's been created is a synthetic table perhaps so it's duplicating rows (?)

Can't seem to get to the bottom of it though

[sunny_talwar]

There is more script then what you have put down here? Because if this is the total script, there is no reason for synthetic key because you will only be left with  All_Profiles as are dropping Table_A and Table_B at the end

[james_butler]

Hi, 

Did you get a solution for this? I've just encountered a similar problem.

Thanks

James