Dynamic schema with some business rules - suggestions please

[nivedhitha]

Hi Everyone,

The above picture shows 4 source - target mappings with similar transformations and similar schema in both source and target tables. I wanna be able to do this using a single job and have an external control table to say which table maps to which target plus any additional column that I may need. So that way, even if a 5th source come in I can add it to the control table and this job could run the new mapping too without any changes in the job. Does anyone have any suggestion on where I should start? I'm so clueless right now.

[fdenis]

you are rewriting a rule engine….
you can do it using a conf file who know all types of rules like regexp, list of coice, ….
then apply you rules step by step

you are going to be between ETL and Java code but it's working fine.

[nivedhitha]

@fdenis ,

Thanks for your reply.

Can you please explain it a bit more detailed? I find it hard to understand what you mean.

[fdenis]

began by list type of rule.
then hard code one rule.
add context to your job tu make it full dynamic.
add a new job who read a file (conf file) ant then run previous job with the corresponding parameters.
it's done.

[nivedhitha]

Hi @fdenis ,

Thanks for that. I have more questions though

1. Where do I define my rules? 

2. what should my config file have?

[fdenis]

define your rules into config file it may be an xml file like:
<Table name="tableName>
<column columnName="col1" type="matchRegexp" regexp="[0-9]*"\>
<column columnName="col1" type="....
…
<\Table>