topic Dynamic Data Mapping and Transformation in Talend Studio

Dynamic Data Mapping and Transformation

Rohitash — Sat, 16 Nov 2024 03:03:06 GMT

Hello Team,

I have one requirement in Talend where I have to do dynamic mapping of multiple files/tables.

Consider I have 5 files with 5 different schema for that I have to used Dynamic Schema.

Now, the challenges is how to dynamically map those 5 different schema.

I don't want to use tMap 5 times in the design.

How to achieve dynamic mapping in talend ?

Kindly assist.

Re: Dynamic Data Mapping and Transformation

JohnRMK — Mon, 09 Mar 2020 18:28:01 GMT

Hello,

Can you give us more details on the need and what you want to do. Is it a migration, loading a data warehouse ... concrete examples of sources and targets.

Re: Dynamic Data Mapping and Transformation

Rohitash — Tue, 10 Mar 2020 05:27:27 GMT

Hi ,

Yes !! It's a data loading from source to target database.

My Requirement is bit complex.

I have 3 different schema and i have to do mapping and transformation dynamically for respective schema, without using multiple times of tMap Component.

Example : Consider I have 3 files

Target Database

1) File : Employee Details

Schema : E_Id | E_Name | E_Sal (dynamically mapping) Employee Table

-----------------------------

2) File : Department Details | TMAP |

Schema : D_Id | D_Name | | Department Table

------------------------------

3) File : Country Details

Schema : C_Code | Country_Name Country Table

In the above 3 different schema , we have to dynamically map the data and load it in the target database.

Kindly assist.

In case any queries , please call me on below mentioned number

Rohitash Sherigar

(M: +91-9594733034)

Re: Dynamic Data Mapping and Transformation

JohnRMK — Tue, 10 Mar 2020 09:27:11 GMT

You will find in this link an example of Talend Job to inject data in different table (or extract them)

https://community.talend.com/yutwg22796/board/message?board.id=integrating&message.id=118027#M118027

Re: Dynamic Data Mapping and Transformation

Rohitash — Tue, 10 Mar 2020 14:35:33 GMT

I have one file which needs to transform using tmap.

I have another file which needs to transform using tmap.

so instead of using multiple times tmap component, I want to use only one tmap component for 2 different file which has different schema.

Scenario

1) In local directory, I have multiple files . To fetch that file , I'll be using tfilelist component to iterate the source file.

2) Each file has different schema, different schema has to map with tmap. which means multiple times tmap component is used. And I don't want to use multiple time tmap component in my job.

So, what is the best way to implement this job design.

I have attached the image of a scenario. Please find it and assist me .

scenario.JPG

Re: Dynamic Data Mapping and Transformation

JohnRMK — Tue, 10 Mar 2020 14:55:56 GMT

From what I understand, it's impossible to do it!

the dynamic schema is used to load columns not known at dev and known at runtime.
With a tMap, you have to act on the name of a well defined column.

Generally the dynamic schema is used at the end of a schema and retrieves all the columns.

Id_1 (int) ==> can be modified in a tMap
f_name (string) ==> can be modified in a tMap
l_name (string) ==> can be modified in a tMap
col_1; col_2; col_3; ....... (dynamic) ==> cannot be modified in tMap.

You can either try to directly transform the file locally or if you put your data in a staging database with the dynamic schema and you apply the transformation with procedures or update query (DML) in the database.

Finally, you load the data into the final database with the dynamic schema again.

If you don't have a lot of transformation, you can use the examples below:
https://help.talend.com/reader/KxVIhxtXBBFymmkkWJ~O4Q/WX1guuvSnxZg5nBBQ26LBQ

https://help.talend.com/reader/hCrOzogIwKfuR3mPf~LydA/CHTC16vtJ5cev_GPSTCO2w