Hi all,
I am doing a mini project which requires me to load 1000+ files from source to target. Each file has a different structure. It is also requires to perform data quality checks on the data before it is loaded into the database. Files are Comma delimited.
My main idea is to not to create 1000+ jobs as it will be not easy to maintain. I am looking for re-usable code that can be used for this purpose. For data quality purpose we are good as we have come up with a framework for the same and it will be done at database level. I am more concerned about loading database tables.
I have already explored following options:
1. Use Dynamic schema using tSetDynamicSchema component but it seems to be working only for the postional files not delimited files.
2. Referred following links but it didn't really help:
http://bekwam.blogspot.in/2011/06/dynamic-schemas-in-talend-open-studio.html
https://www.talend.com/blog/2016/03/11/how-to-operationalize-meta-data-in-talend-with-dynamic-schema...
Can someone please help with it?