Skip to main content
Announcements
Introducing a new Enhanced File Management feature in Qlik Cloud! GET THE DETAILS!
cancel
Showing results for 
Search instead for 
Did you mean: 
Anonymous
Not applicable

Talend Dynamic Schema

Hi all,
I am doing a mini project which requires me to load 1000+ files from source to target. Each file has a different structure. It is also requires to perform data quality checks on the data before it is loaded into the database. Files are Comma delimited. 
My main idea is to not to create 1000+ jobs as it will be not easy to maintain. I am looking for re-usable code that can be used for this purpose. For data quality purpose we are good as we have come up with a framework for the same and it will be done at database level. I am more concerned about loading database tables.
I have already explored following options:
1. Use Dynamic schema using tSetDynamicSchema component but it seems to be working only for the postional files not delimited files.
2. Referred following links but it didn't really help:
http://bekwam.blogspot.in/2011/06/dynamic-schemas-in-talend-open-studio.html
https://www.talend.com/blog/2016/03/11/how-to-operationalize-meta-data-in-talend-with-dynamic-schema...
Can someone please help with it?
Labels (2)
1 Reply
Anonymous
Not applicable
Author

Hi,
Usually, we use  tfilelist to iterate these files and load them in one target.
The work flow looks like: tfilelist(set a filemask )-->iterate>tfileinputdelimited(dynamic schema)-->t<DB>output.
The dynamic schema feature will be available in talend subscription solution which  allows you to design schema with an unknown column structure (unknown name and number of columns).

Best regards
Sabrina