How to handle structure of source tables/files dynamically to load the data into a file format target S3?

[Anonymous]

Hi, 

I want to achieve a situation where I have different sources(tables and files) all are having the different structure. So, need a way to import these sources dynamically or with a metadata driven approach and able to transform the columns as well by applying any business logic before loading into target.

For example:

Source table-1(structure gets changed on daily basis) -->  target file 1

Source file-1(structure gets changed on daily basis) --> target file 2

Kindly suggest how we can achieve this in Talend.

NOTE: I'm aware about the Dynamic column functionality in various Talend components but:

1. How can we transform the columns getting in dynamic column

2. And I'm creating a Spark job and it doesn't support this functionality.

[Anonymous]

Hello,

Dynamic schema type can be used with components in  a DI standard job. But, it is not available with a Big Data Spark job in Talend Studio designer.

Here exists a new feature jira issue:https://jira.talendforge.org/browse/TBD-3302

Best regards

Sabrina