Talend Dynamic Schema

Anonymous — Sat, 16 Nov 2024 10:18:07 GMT

Hi all,
I am doing a mini project which requires me to load 1000+ files from source to target. Each file has a different structure. It is also requires to perform data quality checks on the data before it is loaded into the database. Files are Comma delimited.
My main idea is to not to create 1000+ jobs as it will be not easy to maintain. I am looking for re-usable code that can be used for this purpose. For data quality purpose we are good as we have come up with a framework for the same and it will be done at database level. I am more concerned about loading database tables.
I have already explored following options:
1. Use Dynamic schema using tSetDynamicSchema component but it seems to be working only for the postional files not delimited files.
2. Referred following links but it didn't really help:
http://bekwam.blogspot.in/2011/06/dynamic-schemas-in-talend-open-studio.html
https://www.talend.com/blog/2016/03/11/how-to-operationalize-meta-data-in-talend-with-dynamic-schemas
Can someone please help with it?

Re: Talend Dynamic Schema

Anonymous — Fri, 28 Oct 2016 10:15:18 GMT

Hi,
Usually, we use tfilelist to iterate these files and load them in one target.
The work flow looks like: tfilelist(set a filemask )-->iterate>tfileinputdelimited(dynamic schema)-->t<DB>output.
The dynamic schema feature will be available in talend subscription solution which allows you to design schema with an unknown column structure (unknown name and number of columns).

Best regards
Sabrina

topic Talend Dynamic Schema in Installing and Upgrading

Talend Dynamic Schema

Re: Talend Dynamic Schema