topic Re: How to transform the dynamic column before inserting it into desired target? in Talend Studio

How to transform the dynamic column before inserting it into desired target?

Anonymous — Sat, 16 Nov 2024 07:44:54 GMT

Hi,

I'm getting the data from source in dynamic column. Lets say 4 columns comes in it. So, I want to apply some transform over 2 columns then send the data into target/output.

And want a dynamic solution for achieving this situation like columns will be created dynamically in target side.

Please suggest......

Re: How to transform the dynamic column before inserting it into desired target?

fdenis — Thu, 30 Aug 2018 10:48:03 GMT

how do you identify columns to work on in dynamics?
if you have 4 columns why don't you get 4 columns?

Re: How to transform the dynamic column before inserting it into desired target?

Anonymous — Thu, 30 Aug 2018 13:57:42 GMT

Real time situation is like we are getting a file on daily basis and its structure is changing mostly on daily basis.

For example, on day-1 we receive a file with 4 columns, on day-2 we receive a file with 6 columns(2 new columns added) similarly on day-3 we receive a file with 3 columns only(3 columns removed from file) so on......

So, how I can handle this situation dynamically so that my columns at target(AWS S3) side will get adjusted based on the source file. I don't want to pre-define any schema on both source and target side.

Also, can I also transform the columns that are coming in source file before sending them into target?

Re: How to transform the dynamic column before inserting it into desired target?

fdenis — Thu, 30 Aug 2018 15:33:30 GMT

You can use tJavaFlex to work (using java code) on Dynamic fields.Good luck.

Re: How to transform the dynamic column before inserting it into desired target?

Anonymous — Thu, 30 Aug 2018 19:27:19 GMT

Any help document or material available for using tjavaflex with an example?

Re: How to transform the dynamic column before inserting it into desired target?

Anonymous — Thu, 30 Aug 2018 19:45:19 GMT

Instead of writing java code can i write s python code for transforming the columns ?

Re: How to transform the dynamic column before inserting it into desired target?

fdenis — Fri, 31 Aug 2018 09:12:40 GMT

no, Talent is a java code generator...

Re: How to transform the dynamic column before inserting it into desired target?

fdenis — Fri, 31 Aug 2018 09:28:03 GMT

int ncount=input_row.c.getColumnCount();
DynamicMetadata column0=row1.c.getColumnMetadata(ncount-2);
DynamicMetadata column1=row1.c.getColumnMetadata(ncount-1);
System.out.println("The column name is:"+column0.getName()+" The length of column is: "+column0.getLength()+" The column Type is: "+column0.getType());
System.out.println("The column name is:"+column1.getName()+" The length of column is: "+column1.getLength()+" The column Type is: "+column1.getType());

it's a sample of code but usefull.
Thanks to @shong