How to transform the dynamic column before inserting it into desired target?

[Anonymous]

Hi,

I'm getting the data from source in dynamic column. Lets say 4 columns comes in it. So, I want to apply some transform over 2 columns then send the data into target/output.

And want a dynamic solution for achieving this situation like columns will be created dynamically in target side.

Please suggest......

[fdenis]

how do you identify columns to work on in dynamics?
if you have 4 columns why don't you get 4 columns?

[Anonymous]

Real time situation is like we are getting a file on daily basis and its structure is changing mostly on daily basis.

For example, on day-1 we receive a file with 4 columns, on day-2 we receive a file with 6 columns(2 new columns added) similarly on day-3 we receive a file with 3 columns only(3 columns removed from file) so on......

So, how I can handle this situation dynamically so that my columns at target(AWS S3) side will get adjusted based on the source file. I don't want to pre-define any schema on both source and target side.

Also, can I also transform the columns that are coming in source file before sending them into target?

[fdenis]

You can use tJavaFlex to work (using java code) on Dynamic fields.Good luck.

[Anonymous]

Any help document or material available for using tjavaflex with an example?

[Anonymous]

Instead of writing java code can i write s python code for transforming the columns ?

[fdenis]

no, Talent is a java code generator...

[fdenis]

int ncount=input_row.c.getColumnCount();
DynamicMetadata column0=row1.c.getColumnMetadata(ncount-2);
DynamicMetadata column1=row1.c.getColumnMetadata(ncount-1);
System.out.println("The column name is:"+column0.getName()+" The length of column is: "+column0.getLength()+" The column Type is: "+column0.getType());
System.out.println("The column name is:"+column1.getName()+" The length of column is: "+column1.getLength()+" The column Type is: "+column1.getType());


it's a sample of code but usefull.
Thanks to @shong