topic Re: Dynamic schema with some business rules - suggestions please in Talend Studio

Dynamic schema with some business rules - suggestions please

nivedhitha — Sat, 16 Nov 2024 04:36:43 GMT

Hi Everyone,

The above picture shows 4 source - target mappings with similar transformations and similar schema in both source and target tables. I wanna be able to do this using a single job and have an external control table to say which table maps to which target plus any additional column that I may need. So that way, even if a 5th source come in I can add it to the control table and this job could run the new mapping too without any changes in the job. Does anyone have any suggestion on where I should start? I'm so clueless right now.

Re: Dynamic schema with some business rules - suggestions please

fdenis — Fri, 20 Sep 2019 16:07:37 GMT

you are rewriting a rule engine….
you can do it using a conf file who know all types of rules like regexp, list of coice, ….
then apply you rules step by step

you are going to be between ETL and Java code but it's working fine.

Re: Dynamic schema with some business rules - suggestions please

nivedhitha — Fri, 20 Sep 2019 16:18:59 GMT

@fdenis ,

Thanks for your reply.

Can you please explain it a bit more detailed? I find it hard to understand what you mean.

Re: Dynamic schema with some business rules - suggestions please

fdenis — Mon, 23 Sep 2019 09:52:41 GMT

began by list type of rule.
then hard code one rule.
add context to your job tu make it full dynamic.
add a new job who read a file (conf file) ant then run previous job with the corresponding parameters.
it's done.

Re: Dynamic schema with some business rules - suggestions please

nivedhitha — Mon, 23 Sep 2019 15:35:20 GMT

Hi @fdenis ,

Thanks for that. I have more questions though

1. Where do I define my rules?

2. what should my config file have?

Re: Dynamic schema with some business rules - suggestions please

fdenis — Mon, 23 Sep 2019 16:31:26 GMT

define your rules into config file it may be an xml file like:
<Table name="tableName>
<column columnName="col1" type="matchRegexp" regexp="[0-9]*"\>
<column columnName="col1" type="....
…
<\Table>