Dynamic Data Mapping and Transformation

[Rohitash]

Hello Team,

I have one requirement in Talend where I have to do dynamic mapping of multiple files/tables.

Consider I have 5 files with 5 different schema for that I have to used Dynamic Schema.

Now, the challenges is how to dynamically map those 5 different schema.

I don't want to use tMap 5 times in the design.

How to achieve dynamic mapping in talend ?

Kindly assist.

[JohnRMK]

Hello,

Can you give us more details on the need and what you want to do. Is it a migration, loading a data warehouse ... concrete examples of sources and targets.

[Rohitash]

Hi ,

Yes !! It's a data loading from source to target database.

My Requirement is bit complex.

I have 3 different schema and i have to do mapping and transformation dynamically for respective schema, without using multiple times of tMap Component.

Example : Consider I have 3 files

                                                                                                                                  Target  Database

1) File : Employee Details                                                                                           

Schema :  E_Id | E_Name | E_Sal        (dynamically mapping)                                 Employee Table

                                                            -----------------------------

2) File : Department Details                |           TMAP            |     

Schema : D_Id | D_Name                   |                                 |                                    Department Table

                                                           ------------------------------

3) File : Country Details 

Schema : C_Code | Country_Name                                                                          Country Table

In the above 3 different schema , we have to dynamically map the data and load it in the target database.

Kindly assist. 

In case any queries , please call me on below mentioned number

Rohitash Sherigar

(M: +91-9594733034)

[JohnRMK]

You will find in this link an example of Talend Job to inject data in different table (or extract them) 

https://community.talend.com/yutwg22796/board/message?board.id=integrating&message.id=118027#M118027

[Rohitash]

I have one file which needs to transform using tmap. 

I have another file which needs to transform using tmap.

so instead of using multiple times tmap component, I want to use only one tmap component for 2 different file which has different schema.

Scenario

1) In local directory, I have multiple files . To fetch that file , I'll be using tfilelist component to iterate the source file.

2) Each file has different schema, different schema has to map with tmap. which means multiple times tmap component is used. And I don't want to use multiple time tmap component in my job.

So, what is the best way to implement this job design.

I have attached the image of a scenario. Please find it and assist me .

scenario.JPG

[JohnRMK]

From what I understand, it's impossible to do it!

the dynamic schema is used to load columns not known at dev and known at runtime.
With a tMap, you have to act on the name of a well defined column.

Generally the dynamic schema is used at the end of a schema and retrieves all the columns.

Id_1 (int) ==> can be modified in a tMap
f_name (string) ==> can be modified in a tMap
l_name (string) ==> can be modified in a tMap
col_1; col_2; col_3; ....... (dynamic) ==> cannot be modified in tMap.

You can either try to directly transform the file locally or if you put your data in a staging database with the dynamic schema and you apply the transformation with procedures or update query (DML) in the database.

Finally, you load the data into the final database with the dynamic schema again.

If you don't have a lot of transformation, you can use the examples below:
https://help.talend.com/reader/KxVIhxtXBBFymmkkWJ~O4Q/WX1guuvSnxZg5nBBQ26LBQ

https://help.talend.com/reader/hCrOzogIwKfuR3mPf~LydA/CHTC16vtJ5cev_GPSTCO2w