topic Re: how to prevent duplicate records from inserting in Talend Studio

how to prevent duplicate records from inserting

Anonymous — Sat, 16 Nov 2024 07:26:04 GMT

HI,

I have a csv file which gives me source to target mapping.

ex:

source;month;status;target
AB1;01;A;A01
AB2;02;B;B02
AB2;02;C;C02

For every unique value for source,month& status, target value is given.

Since this file is manually updated. I want to write a logic that in case of same(duplicate) entries inserted for these 3 columns, the file should get rejected. how do i do it.?

This is how my job looks..? tfileinput is my input csv file. in Tmap, i just uppercase all the fields.

So if duplicate entries are inserted in csv file.my file should be rejected.. Thanks in advance!

Re: how to prevent duplicate records from inserting

TRF — Sun, 21 Oct 2018 22:07:58 GMT

Add a tUniqRow after the tFileInputDelimited then you can check the value of the variable tUniqRow_1_NB_DUPLICATES.
I suggest you to add a tJava connected to the tFileInputDelimited by a onSubjobOk trigger. Don't change anything to this component and connect it to a tFileDelete with If trigger with the following condition (Integer)globalMap.get("tUniqRow_1_NB_DUPLICATES") > 0

Re: how to prevent duplicate records from inserting

Anonymous — Sun, 21 Oct 2018 23:16:21 GMT

Hi @TRF, thanks for the reply.

I dont to delete the file in case a duplicate is found. I just want the load to output to get error out.

also, can u pls do dummy job to let me know how it is done.? I'm not sure if i m doing right, as it is not populating results.

The java component gives me a msg - if the output has components , then there must b input link to propagate data.

Re: how to prevent duplicate records from inserting

TRF — Mon, 22 Oct 2018 07:57:15 GMT

1- you forgot the tUniqRow to get duplicate records

2- you have used a tJavaRow instead of a tJava