Need to dedup input data on the basis of number of occurence of a particular column

[mgoel]

input - CSV file

this csv file has columns C1, c2, c3,c4 and all are strings

I want to dedup it on the basis of below logic -

apply dedup on all rows on the basis of column C2, keep all non-duplicate values.

on the duplicate ones, keep those values where C1 == "ABC"

than combine both of these results and form an output.

I tried this way (first by counting the occurrence of C2 and later on applied filter on it), but I don't want to read the delimited file multiple times. Kindly help

is there any other shortcut way to do it.

[mgoel]

many times, when I am running my job my tdelimitedfile is not reading anything. not sure why. it can be seen in the snippet

[Anonymous]

Hi

Only read the file one time, store the data into memory using tHashOutut component, read the data back from memory using tHashInput whenever you want, for example:

tfileInputDelimited--main--tHashOutput

|onsubjobok

tHashInput1---tMap.....

-lookup

tHashInput2

Regard

Shong

[gjeremy1617088143]

Hi, tUniqRow component allow you to separate unique and duplicate row in two different flow.You can do the control you want then send both flow in linked tHashOutput.

Then you can read the hash after an onsubjob ok with tHashInput.