[resolved] compare string with a list

[Anonymous]

Hi Everyone
I wanna read string from a CSV file(A) one by one.
And compare each string with string list(CSV B).
If string from A contains the same substring in B, then exclude the substring from A and output.
Ex:
A File
---------------------
oneniceday
luckydog
writeyourmessage
---------------------
B File
---------------------
one
nice
dog
message
---------------------
Processing:
---------------------
oneniceday --> exclude 'one' 'nice' --> day
luckydog --> exlude 'dog' --> lucky
writeyourmessage --> exlude 'message' --> writeyour
---------------------
result:
---------------------
day
lucky
writeyour
---------------------
Is there any suggestion, idea or component to implement?
Appreciated!

[Anonymous · Accepted solution]

Hi
There is no a specialized component to do this, the solution is to read each line from File A, and then iterate each line from File B, and write a piece of Java code remove the same string from File A, the job is showed in my screenshot.
on tJava_1, put the current record from A to a global variable.

globalMap.put("result",row1.a);

on tJava: write a piece of Java code to check if the string from A contains the same substring in B, and remove the substring, for example:

int substring_index=0;
String result=(String)globalMap.get("result");
String b=(String)globalMap.get("row2.b");
if(result.contains(b)){
substring_index=result.indexOf(row2.b);
globalMap.put("result",result.substring(0,substring_index)+result.substring(substring_index+b.length()));
}

on tFixedFlowInput_3, output the end result after all iteration on records from B.
result:

Starting job ddf at 13:52 18/11/2013.

 connecting to socket on port 4026
 connected
day
lucky
writeyour
 disconnected
Job ddf ended at 13:52 18/11/2013. 

Shong

[Anonymous]

Hi
There is no a specialized component to do this, the solution is to read each line from File A, and then iterate each line from File B, and write a piece of Java code remove the same string from File A, the job is showed in my screenshot.
on tJava_1, put the current record from A to a global variable.

globalMap.put("result",row1.a);

on tJava: write a piece of Java code to check if the string from A contains the same substring in B, and remove the substring, for example:

int substring_index=0;
String result=(String)globalMap.get("result");
String b=(String)globalMap.get("row2.b");
if(result.contains(b)){
substring_index=result.indexOf(row2.b);
globalMap.put("result",result.substring(0,substring_index)+result.substring(substring_index+b.length()));
}

on tFixedFlowInput_3, output the end result after all iteration on records from B.
result:

Starting job ddf at 13:52 18/11/2013.

 connecting to socket on port 4026
 connected
day
lucky
writeyour
 disconnected
Job ddf ended at 13:52 18/11/2013. 

Shong

Millions of thanks
You are the best Shong!

[Anonymous]

Hi,
Could you explain the last 3 steps in the code above
started with
if(result.contains(b))
Thanks in advance.

[Anonymous]

Hi Shong,

Why can't we use the tmap and exp filter within that?
Actually i have the similar requirement but when i do that i don't get the exact out put.
1> file A has the main data
2>File B has the list of IDs which if present in the File A has to be removed
3> i used the regular tmap and the used the !contains(b)
Result:
Lets say 5 in file A and 4 in file B, and suppose all 4 have matching in B
Expected output: 1
Actual out i am getting:
(5*4) =20 -1 =19 and 1 goes to reject row
so its like whatever i dont want goes 1 single file but waht ever i want + what i dont want goes to other file, i have tried all setting but this work more or less same way..

Any help will be app!!