topic Re: Apply logic to all fields at once? in Talend Studio

Apply logic to all fields at once?

the_etl_guy — Tue, 20 Feb 2018 02:33:24 GMT

Hello everyone,

Is there a way to apply some logic to all fields at once?

Like if I have to apply LTRIM to all 200 incoming fields then instead of doing this at individual field level i.e 200 times, is there any not so laborious way to achieve this?

Re: Apply logic to all fields at once?

manodwhb — Tue, 20 Feb 2018 04:35:41 GMT

You may need to call manully ,untill unless you call the specific funaction,it will not do the required operation.

Re: Apply logic to all fields at once?

the_etl_guy — Tue, 20 Feb 2018 05:24:13 GMT

Thanks Manohar, could you please elaborate?

Re: Apply logic to all fields at once?

ashif2 — Tue, 20 Feb 2018 05:32:38 GMT

We can reduce our manual intervention by some tricky way (below is my thought process but there could be some other easy also)

Create a routine with ltrim functionality with null handling scenario .

your input ---> tjava row (sync the columns , generate the code ) and with the help of text editor 's find and replace option apply the newly created routine ..

Sample Routine:

package routines;

public class ltrimr {
public static String ltrim(String datain) {
String dataout="";
if (datain == null) {
dataout= datain;
}
else if(datain!=null)
{

dataout= datain.replaceAll("^\\s+","");
}
return dataout;

}
}

tjava row logic :

output_row.newColumn = ltrimr.ltrim(input_row.newColumn);
output_row.newColumn1 = ltrimr.ltrim(input_row.newColumn1);
output_row.newColumn2 = ltrimr.ltrim(input_row.newColumn2);

Re: Apply logic to all fields at once?

the_etl_guy — Tue, 20 Feb 2018 05:55:47 GMT

Thanks @ashif, that's an interesting approach when used with a text editor. I am also thinking of using a database function on each field at SQL level, again a text editor will save a lot time here but am not sure how it will affect the performance.

SELECT

A,B,C,D.....

FROM TABLE;

REPLACE "," with "),LTRIM(" will do 99% of the job in a jiffy!

SELECT
A),LTRIM(B),LTRIM(C),LTRIM(D.....
FROM TABLE;

But again, this is handy and efficient only in case of simple operations like LTRIM

Re: Apply logic to all fields at once?

ashif2 — Tue, 20 Feb 2018 06:03:47 GMT

it could be better , if you explain the scenario with complete information like nature of source (whether it is relational,structured files, semi- structured files ). Nothing has been specified so community people will try to give you a generic solution (solution at talend level).

Re: Apply logic to all fields at once?

the_etl_guy — Tue, 20 Feb 2018 06:15:53 GMT

yeah.. well the source is MSSQL. Table has 200 fields and the issue is that any string field can have "\n" as a hidden character. This causes a row in target file to break into multiple rows. To overcome this, I have to replace "\n" with "". The problem is that "\n" can appear in any field so the logic needs to applied on all fields.

Re: Apply logic to all fields at once?

ashif2 — Tue, 20 Feb 2018 06:55:21 GMT

Create a routine with logic replaceAll("\\r\\n|\\r|\\n", " ") ( include null handling logic) and call the routine in tJavaRow. Its handy because if any other new characters needs to be handled which can be achieved by simple change and routine and regeneration of code-build

Re: Apply logic to all fields at once?

the_etl_guy — Tue, 20 Feb 2018 22:25:28 GMT

@ashif I get your point, probably this is the easiest way so far. Thanks!