Dynamic job to load multiple tables with different structure in one talend job

[saivsr]

Hi All,
I have one scenario,

I want to load all tables of data in to respective target tables through one talend job dynamically..please help me on this.

like

src1
-------
database--db1
schema---x
tables-----A,B,C,D,E,F,........X,Y,Z.                                                                     Tareget
all tables are different schema structure.                                                          ---------------------------
                                                                                                                          database----------db3
                                                                                                                           schema------------z
src2                                                                                                                    tables------------A,B,C,D,E,F,..........X,Y,Z.
---------                                                                               These tables schema structure same as src1,src2 tables structure
database---db2
schema--y
tables are same like src1---A,B,C,D,E,F,.............X,Y,Z.
all tables are diffrerent schema structure

i want load union the data of src1,src2 tables data then load to target respective tables in single talend job dynamically for all tables.

[saivsr]

my source and target databases are MSSQL...I want to create one talend job instead of creating multiple jobs for all tables..

Please suggest me.Thanks..all

[saivsr]

Hi guys the above scenario like as, 
I have 100 tables with different metadata and i need to develop a single job with loads all these tables to target system we Need to run 10 times for all tables or at a time if so please guide me out.

SRC----MSSQL

TARGET---MSSQL

[Anonymous]

Use dynamic column type.  However, this only works if you do not need to do transformation, since you do not know the schema and data type upfront.

Else use the Template functionality in the Studio to generate all your jobs https://help.talend.com/reader/itosd4jFDXxrm6qULuPrbg/rpFV93Uc4LKksvXQFtSdWg  It will generate many jobs, but it take 5 mins instead of several days.

[saivsr]

Hello Iburatally Employe Thanks for you response.

I am using Talend Open Studio..So in this there is no option for "Create job from templates".

Please give some solutuion for TOS.

I created source tables passing iteratively.

Pleas find attached screen shot.

talend_dynamic_job.PNG

[shivanand]

Hi ,

You can achieve this using dynamic column load. Following are the steps

1.  Create a list of your table names in one csv file 

2. Read that file in tFileInputDelimited

3. Use tFlowToIterate  and create global variable in with tablenames as value from file

4. Use tmysqlinput and set table name as global variable name and in schema select as built in and set only one row and select type as dynamic .

5. do same for another database input flow

6. use tunite to make union 

7. connect to target where specify table name as global variable from tFlowToIterate  and schema as built in and then create only one row in schema with type as dynamic.

Key here is to pass table names list using tFlowToIterate   and defining dynamic column  in built in schema

You can use following link for reference to dynamic column 

https://help.talend.com/reader/OnsdPcJL19xPfNBaWRUL~w/6uAF9tc63_edATsd7VMTwg

Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.

[saivsr]

Hi Shivanand,

Thanks for you response..

I am using Talend Open Studio(TOS)..In this there is no option dynamic rt?

How i can ..? Please 

[muralam]

Hi could you please elaborate more on the job, 1.tfileinputdelimited I listed all the table names
2.tflowtoiterate defined a global variable there
3.taccessinput in table name I am passing the global variable where it get table name from step 1 here where I am stuck job is throwing an error unexpected token

[Anonymous]

Hi All,

Can anyone please help me here?

My requirement is to load data from MySQL/Oracle to Hive.

1) I want to load multiple tables with different structure in one talend job.

2) I am using Talend open studio so not having feature of Dynamic datatype.

3) Is there any way to achieve it in talend open studio version?

Thanks