topic Re: Data ingestion talend job in Talend Studio

Data ingestion talend job

singhg — Sat, 16 Nov 2024 06:22:30 GMT

I am working on developing a single talend job for data ingestion from any source to any target which satisfies the below requirement:-

1.) The source and target table names will be specified in the context variables before running the job.

2.) It should read the source and target schema (like table name, column name, data type, primary key ) on the go from a database table.

2.) It should perform full load, incremental load based upon primary key, datetime for data ingestion.

Kindly let me know, how can I read source/target schema on go, create source target components, map the source/target columns, perform transformation for full/incremental load.

Is this even feasible or not. Please let me know.

Re: Data ingestion talend job

fdenis — Mon, 11 Mar 2019 10:49:24 GMT

hi,
there is no dynamic schémas on the free version of Talend.
so the best way for you is to make one job by table.¨
an other possibility is to get all data on one String (it's ok for small tables).
concatenate all fields on select query to create the insert query.

if all your table have unique key you can use it to copy data field by fields.

good luck

you can use system view or talbes to discover db schemas.

Re: Data ingestion talend job

singhg — Tue, 12 Mar 2019 02:12:14 GMT

Since I will be using incremental load along with dynamic schema, is there any way to perform lookup on tables using dynamic schema in tmap.

Re: Data ingestion talend job

fdenis — Tue, 12 Mar 2019 09:08:15 GMT

yes but you have to extract field value by name so you have to know it. it cannot be dynamic.
if you know the name of the fiead you can get it from your SQL query and let other filds dynamics.

Re: Data ingestion talend job

fdenis — Tue, 12 Mar 2019 09:10:17 GMT

think about system table to get the schema of table...

Re: Data ingestion talend job

singhg — Tue, 12 Mar 2019 10:24:23 GMT

yes, for schema detail, we have created another table which contains schema details for sources and targets

like table name, column name, datatype, length of datatype, primary key etc.