topic Re: Searching distincrt elements in a line in Talend Studio

Searching distincrt elements in a line

trimech — Sun, 01 Mar 2020 13:15:39 GMT

Hello evryone,

I need your help.

I have a csv file with this 3 lines :

a;b;b;c;d;d;v;a;g
f;f;a;g;a;g;h;p
a;a;a;a;a;a;a;b

As a result, i would like to have a distinct lines :

a;b;c;d;v;g

f;a;g;h;p

a;b

Thanks for your contribution

Re: Searching distincrt elements in a line

Anonymous — Mon, 02 Mar 2020 13:59:29 GMT

This is an interesting requirement as you are essentially asking to treat columns in a similar way to which you would treat rows. However, I have knocked up a quick example that appears to work. It will limit occurrences of strings to once per row and will sort them alphabetically. A screenshot of the job can be seen below.....

The bit you need to look at is the code in the tJavaFlex. Pay attention to the fact that I have used default column names and I have unticked the "Data Auto Propagate" option. The code is in the Main Code section and looks like this.....

//Code generated according to input schema and output schema
String[] list = new String[9];

list[0] = row1.newColumn;
list[1] = row1.newColumn1;
list[2] = row1.newColumn2;
list[3] = row1.newColumn3;
list[4] = row1.newColumn4;
list[5] = row1.newColumn5;
list[6] = row1.newColumn6;
list[7] = row1.newColumn7;
list[8] = row1.newColumn8;

for(int i = 0; i<9; i++){
	for(int x = 0; x<9; x++){
		if(list[i].equals(list[x]) && i!=x){
			list[i]="";
		}
	}

}

//Sort array alphabetically
List<String> storedValues = java.util.Arrays.asList(list);
java.util.Collections.sort(storedValues);

//Return values to original array
list = storedValues.toArray(list);

//Remove empty Strings from array
String[] cleaned_list = new String[9];

int pos = 0;

for(int i=0; i<list.length;i++){
	if(!list[i].equals("")){
		cleaned_list[pos] = list[i];
		pos++;
	}

}


//Assign ordered, distinct values
row2.newColumn = cleaned_list[0];
row2.newColumn1 = cleaned_list[1];
row2.newColumn2 = cleaned_list[2];
row2.newColumn3 = cleaned_list[3];
row2.newColumn4 = cleaned_list[4];
row2.newColumn5 = cleaned_list[5];
row2.newColumn6 = cleaned_list[6];
row2.newColumn7 = cleaned_list[7];
row2.newColumn8 = cleaned_list[8];

The input data I used was as follows....

a;b;b;c;d;d;v;a;g
f;f;a;g;a;g;h;p
a;a;a;a;a;a;a;b

....the data returned looked like this....

a|b|c|d|g|v|||
a|f|g|h|p||||
a|b|||||||

Re: Searching distincrt elements in a line

uzix — Mon, 02 Mar 2020 15:13:02 GMT

hello,

indeed as @rhall said is an interesting requirement , i present an alternate solution.

i use a tFileInputFullRow as source for the file with the 3 lines.

connect to a tjavarow - hip1.jpg

use function as in hip2.jpg -> function provided by @shong in previous example to hadle duplicate data in string with a small tweak from me.

.-----------.
| logoutput |
|=---------=|
|line |
|=---------=|
|a;b;c;d;g;v|
|a;f;g;h;p |
|a;b |
'-----------'

[statistics] disconnected

hip2.jpg
hip1.jpg

Re: Searching distincrt elements in a line

Anonymous — Mon, 02 Mar 2020 21:30:26 GMT

Nice solution @uzix, but it might be a bit more tempting for someone to use if you provided the code to be copied 😉 Would you mind sharing the code as text to make this easy for people?

Re: Searching distincrt elements in a line

uzix — Tue, 03 Mar 2020 08:51:46 GMT

hello @rhall , sure here goes:

i just adapted code provided by @shong

public static String removeDupsInPlace(String word) {
final StringBuilder output = new StringBuilder();
String wordSorted = "";
String output_a ="";
char tempArray[] = word.replaceAll(";","").toCharArray();
// sort tempArray
Arrays.sort(tempArray);
wordSorted = new String(tempArray);
for (int i = 0; i < wordSorted.length(); i++)
{
String character = wordSorted.substring(i, i + 1);
if (output.indexOf(character) < 0 ) // if not contained
//output.append(character);
output.append(character).append(";");
}
//remove ; if last char
if(output.toString().endsWith(";"))
{
output_a = output.toString().substring(0,output.length() - 1);
}else{
output_a = output.toString();
}
return output_a;
}

Re: Searching distincrt elements in a line

Anonymous — Tue, 03 Mar 2020 10:11:09 GMT

Thanks @uzix. It makes it so much easier when you can copy and paste code 🙂