tFileInputDelimited-tSetGlobalVar-tMap/tFilterRow depending of preceed

[_AnonymousUser]

hi,
wonder how to manage a row filter which let me group rows depending of field content equality.
in simply java it would be so simple but how i succeed with talend:
globalMap.put("Ship_Number", ""); // can do this with tSetGlobalVar
if (!row1.Ship_Number.equals(globalMap.get("Ship_Number")) ... // if NOT equal do something e.g. an entry in tLogRow or any output file. i can do this with a filter in tMap
globalMap.put("Ship_Number", row1.Ship_Number); // but where can i do this for each row like iterating tSetGlobalVar but within tFileInputDelimited?
hope it's clear enough.
12345 > make new record
12345 > do nothing
22222 > make new record
22222 > do nothing
22222 > do nothing
33333 > make new record
44444 > make new record and so on
thanks in advance

[Anonymous]

Hello

globalMap.put("Ship_Number", row1.Ship_Number); // but where can i do this for each row like iterating tSetGlobalVar but within tFileInputDelimited?

Why do you mean within tFileInputDelimited? I think you can link it to a tJavaRow and filter it on it, eg:
tFileInputDelimited---main--tJavaRow
on tJavaRow, some code like this:

//here the data type of Ship_Number is string.
if((String)globalMap.get("Ship_Number").equals(input_row.Ship_Number)){
do something;
}esle{
do nothing;
}
globalMap.put("Ship_Number",input_row.Ship_Number);

Best regards

shong

[_AnonymousUser]

@shong
thanks very much for your worth hint.
tFileInputDelimited --- row1 (Main) --> tJavaFlex --- row3 (Main) --> tMap

in the end I used an unused field in a tJavaFlex component (Schema Type of tFileInputDelimited) to filter unwanted rows in the tMap with:
row3.Unused_Field.equals("OK")
but I think there is a better solution, isn't it?
my code in tJavaFlex:
// start part of your Java code
java.util.Set<String> ship_numbers = new java.util.HashSet<String>();
// here is the main part of the component,
// a piece of code executed in the row
// loop
if (row1.Ship_Number.trim().length() > 0 &&
ship_numbers.add(row1.Ship_Number)) {
row1.Unused_Field = "OK";
System.out.println("Ship_Number: " + row1.Ship_Number + ' ' + row1.LI_Item_Code);
} else {
System.out.println("Duplicate detected: " + row1.Ship_Number);
}

[Anonymous]

Have you looked at tUniqRow?

[_AnonymousUser]

Have you looked at tUniqRow?

@jholman
Thanks, this seems to be the solution. Will give it a try definitely.
Hope that all duplicates will be removed and not only "the first encountered duplicate".
Instead of tJavaFlex I probably could have used a user defined function in the expression filter of tMap.