Apply logic to all fields at once?

[the_etl_guy]

Hello everyone,

Is there a way to apply some logic to all fields at once?

Like if I have to apply LTRIM to all 200 incoming fields then instead of doing this at individual field level i.e 200 times, is there any not so laborious way to achieve this?

[manodwhb]

You may need to call manully ,untill unless you call the specific funaction,it will not do the required operation.

[the_etl_guy]

Thanks Manohar, could you please elaborate?

[ashif2]

We can reduce our manual intervention by some  tricky way (below is my thought process but there could be some other easy also)

Create a routine with ltrim functionality with null handling scenario .

your input ---> tjava row (sync the columns , generate the code ) and with the help of text editor 's find and replace option apply the newly created routine .. 

Sample Routine:

package routines;

public class ltrimr {
public static String ltrim(String datain) {
String dataout="";
if (datain == null) {
dataout= datain;
}
else if(datain!=null)
{

dataout= datain.replaceAll("^\\s+","");
}
return dataout;

}
}

tjava row logic : 

output_row.newColumn = ltrimr.ltrim(input_row.newColumn);
output_row.newColumn1 = ltrimr.ltrim(input_row.newColumn1);
output_row.newColumn2 = ltrimr.ltrim(input_row.newColumn2);

[the_etl_guy]

Thanks @ashif, that's an interesting approach when used with a text editor. I am also thinking of using a database function on each field at SQL level, again a text editor will save a lot time here but am not sure how it will affect the performance.

SELECT

A,B,C,D.....

FROM TABLE;

REPLACE "," with "),LTRIM(" will do 99% of the job in a jiffy!

SELECT
A),LTRIM(B),LTRIM(C),LTRIM(D.....
FROM TABLE;

But again, this is handy and efficient only in case of simple operations like LTRIM

[ashif2]

it could be better , if you explain the scenario with complete information like nature of source (whether it is relational,structured files, semi- structured files ).  Nothing has been specified so community people will try to give you a generic solution (solution at talend level). 

[the_etl_guy]

yeah.. well the source is MSSQL. Table has 200 fields and the issue is that any string field can have "\n" as a hidden character. This causes a row in target file to break into multiple rows. To overcome this, I have to replace "\n" with "". The problem is that "\n" can appear in any field so the logic needs to applied on all fields.

[ashif2]

Create a routine with  logic replaceAll("\\r\\n|\\r|\\n", " ") ( include null handling logic) and call the routine in tJavaRow.  Its handy because if any other new characters needs to be handled which can be achieved by simple change and routine and regeneration of code-build

[the_etl_guy]

@ashif I get your point, probably this is the easiest way so far. Thanks!