how to prevent duplicate records from inserting

[Anonymous]

HI,

I have a csv file which gives me source to target mapping. 

ex: 

source;month;status;target
AB1;01;A;A01
AB2;02;B;B02
AB2;02;C;C02

For every unique value for source,month& status, target value is given.

Since this file is manually updated. I want to write a logic that in case of  same(duplicate) entries inserted for these 3 columns, the file should get rejected. how do i do it.?

This is how my job looks..? tfileinput is my input csv file. in Tmap, i just uppercase all the fields.

So if duplicate entries are inserted in csv file.my file should be rejected.. Thanks in advance!

[TRF]

Add a tUniqRow after the tFileInputDelimited then you can check the value of the variable tUniqRow_1_NB_DUPLICATES.
I suggest you to add a tJava connected to the tFileInputDelimited by a onSubjobOk trigger. Don't change anything to this component and connect it to a tFileDelete with If trigger with the following condition (Integer)globalMap.get("tUniqRow_1_NB_DUPLICATES") > 0

[Anonymous]

Hi @TRF, thanks for the reply.

I dont to delete the file in case a duplicate is found. I just want the load to output to get error out.

also, can u pls do dummy job to let me know how it is done.? I'm not sure if i m doing right, as it is not populating results.

The java component gives me a msg - if the output has components , then there must b input link to propagate data.

[TRF]

1- you forgot the tUniqRow to get duplicate records

2- you have used a tJavaRow instead of a tJava