topic Dynamic job to load multiple tables with different structure in one talend job in Talend Studio

Dynamic job to load multiple tables with different structure in one talend job

saivsr — Sat, 23 Sep 2017 16:16:12 GMT

Hi All,
I have one scenario,

I want to load all tables of data in to respective target tables through one talend job dynamically..please help me on this.

src1
-------
database--db1
schema---x
tables-----A,B,C,D,E,F,........X,Y,Z. Tareget
all tables are different schema structure. ---------------------------
database----------db3
schema------------z
src2 tables------------A,B,C,D,E,F,..........X,Y,Z.
--------- These tables schema structure same as src1,src2 tables structure
database---db2
schema--y
tables are same like src1---A,B,C,D,E,F,.............X,Y,Z.
all tables are diffrerent schema structure

i want load union the data of src1,src2 tables data then load to target respective tables in single talend job dynamically for all tables.

Re: Dynamic job to load multiple tables with different structure in one talend job

saivsr — Sun, 24 Sep 2017 03:13:15 GMT

my source and target databases are MSSQL...I want to create one talend job instead of creating multiple jobs for all tables..

Please suggest me.Thanks..all

Re: Dynamic job to load multiple tables with different structure in one talend job

saivsr — Sun, 24 Sep 2017 03:45:26 GMT

Hi guys the above scenario like as,
I have 100 tables with different metadata and i need to develop a single job with loads all these tables to target system we Need to run 10 times for all tables or at a time if so please guide me out.

SRC----MSSQL

TARGET---MSSQL

Re: Dynamic job to load multiple tables with different structure in one talend job

Anonymous — Mon, 25 Sep 2017 10:24:56 GMT

Use dynamic column type. However, this only works if you do not need to do transformation, since you do not know the schema and data type upfront.

Else use the Template functionality in the Studio to generate all your jobs https://help.talend.com/reader/itosd4jFDXxrm6qULuPrbg/rpFV93Uc4LKksvXQFtSdWg It will generate many jobs, but it take 5 mins instead of several days.

Re: Dynamic job to load multiple tables with different structure in one talend job

saivsr — Mon, 25 Sep 2017 12:19:37 GMT

Hello Iburatally Employe Thanks for you response.

I am using Talend Open Studio..So in this there is no option for "Create job from templates".

Please give some solutuion for TOS.

I created source tables passing iteratively.

Pleas find attached screen shot.

talend_dynamic_job.PNG

Re: Dynamic job to load multiple tables with different structure in one talend job

shivanand — Mon, 25 Sep 2017 15:10:47 GMT

Hi ,

You can achieve this using dynamic column load. Following are the steps

1. Create a list of your table names in one csv file

2. Read that file in tFileInputDelimited

3. Use tFlowToIterate and create global variable in with tablenames as value from file

4. Use tmysqlinput and set table name as global variable name and in schema select as built in and set only one row and select type as dynamic .

5. do same for another database input flow

6. use tunite to make union

7. connect to target where specify table name as global variable from tFlowToIterate and schema as built in and then create only one row in schema with type as dynamic.

Key here is to pass table names list using tFlowToIterate and defining dynamic column in built in schema

You can use following link for reference to dynamic column

https://help.talend.com/reader/OnsdPcJL19xPfNBaWRUL~w/6uAF9tc63_edATsd7VMTwg

Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.

Re: Dynamic job to load multiple tables with different structure in one talend job

saivsr — Tue, 26 Sep 2017 12:16:51 GMT

Hi Shivanand,

Thanks for you response..

I am using Talend Open Studio(TOS)..In this there is no option dynamic rt?

How i can ..? Please

Re: Dynamic job to load multiple tables with different structure in one talend job

muralam — Tue, 06 Feb 2018 05:46:04 GMT

Hi could you please elaborate more on the job, 1.tfileinputdelimited I listed all the table names
2.tflowtoiterate defined a global variable there
3.taccessinput in table name I am passing the global variable where it get table name from step 1 here where I am stuck job is throwing an error unexpected token

Re: Dynamic job to load multiple tables with different structure in one talend job

Anonymous — Thu, 19 Sep 2019 08:18:26 GMT

Hi All,

Can anyone please help me here?

My requirement is to load data from MySQL/Oracle to Hive.

1) I want to load multiple tables with different structure in one talend job.

2) I am using Talend open studio so not having feature of Dynamic datatype.

3) Is there any way to achieve it in talend open studio version?

Thanks