Data ingestion talend job

[singhg]

I am working on developing a single talend job for data ingestion from any source to any target which satisfies the below requirement:-

1.) The source and target table names will be specified in the context variables before running the job.

2.) It should read the source and target schema (like table name, column name, data type, primary key ) on the go from a database table. 

2.) It should perform full load, incremental load based upon primary key, datetime for data ingestion.

Kindly let me know, how can I read source/target schema on go, create source target components, map the source/target columns, perform transformation for full/incremental load.

Is this even feasible or not. Please let me know.

[fdenis]

hi,
there is no dynamic schémas on the free version of Talend.
so the best way for you is to make one job by table.¨
an other possibility is to get all data on one String (it's ok for small tables).
concatenate all fields on select query to create the insert query.

if all your table have unique key you can use it to copy data field by fields.

good luck

you can use system view or talbes to discover db schemas.

[singhg]

Since I will be using incremental load along with dynamic schema, is there any way to perform lookup on tables using dynamic schema in tmap.

[fdenis]

yes but you have to extract field value by name so you have to know it. it cannot be dynamic.
if you know the name of the fiead you can get it from your SQL query and let other filds dynamics.

[fdenis]

think about system table to get the schema of table...

[singhg]

yes, for schema detail, we have created another table which contains schema details for sources and targets

like table name, column name, datatype, length of datatype, primary key etc.