Hi guys, I am trying to create dynamic job to validating many different data sources. At first, each data source is being pulled from relative folder and trigger data preparation (dynamic - preparation is being started by the name of the folder). All invalid rows are being send to data stewardship. The wanted result is to write the valid and invalid rows of data source to the same file.
I am faceing few issues in thia design: 1. When inserting data to data stewardship, the information about the file name and direcorty is being lost. I need it in order to write to the same file at destination folder. How can i keep it? 2. I would like to split campaigns to folder (many campaigns can be in the same folder- all those campaigns describe different schema of the same source , just like in data preparation). How can i do it? 3. For each preparation i would like to define schema of campaign in data stewardship automatically - save time in inserting same schema at two different places. Also in the stewardship, defining schema is made manually. Is there a solution out there?
