How to handle structure of source tables/files dynamically to load the data into a file format target S3?

Anonymous — Sat, 16 Nov 2024 07:42:52 GMT

Hi,

I want to achieve a situation where I have different sources(tables and files) all are having the different structure. So, need a way to import these sources dynamically or with a metadata driven approach and able to transform the columns as well by applying any business logic before loading into target.

For example:

Source table-1(structure gets changed on daily basis) --> target file 1

Source file-1(structure gets changed on daily basis) --> target file 2

Kindly suggest how we can achieve this in Talend.

NOTE: I'm aware about the Dynamic column functionality in various Talend components but:

1. How can we transform the columns getting in dynamic column

2. And I'm creating a Spark job and it doesn't support this functionality.

Re: How to handle structure of source tables/files dynamically to load the data into a file format target S3?

Anonymous — Thu, 06 Sep 2018 09:00:38 GMT

Hello,

Dynamic schema type can be used with components in a DI standard job. But, it is not available with a Big Data Spark job in Talend Studio designer.

Here exists a new feature jira issue:https://jira.talendforge.org/browse/TBD-3302

Best regards

Sabrina

topic How to handle structure of source tables/files dynamically to load the data into a file format target S3? in Talend Studio

How to handle structure of source tables/files dynamically to load the data into a file format target S3?

Re: How to handle structure of source tables/files dynamically to load the data into a file format target S3?