Dynamic Schemas & Tmaps

[gurn]

Hi

I have a requirement to push data from a SQL Input component dynamically into a tmap /tjavarow and onto a dataPrepRun component.

I have 10s of data sources and ideally, I'd like to build a single pipeline that reads any of the input tables, dynamically sets up the mapping based on the source data and then pushes this data through to the dataprep component. I'm exploring the use of Dynamic data type to read from the DB and then use tjavarow to grab the names of the columns I want to pass to tDatarepRun.

Has anyone performed anything similar - the biggest challenge at the moment is how to generate the output schema I need to propagate mapping dynamically down to the tDataprepRun component.

Any thoughts on this would be great .

Thanks

Dave

[Anonymous]

Hi Dave

If each table has different structure, you need to use the Dynamic schema, and set a dynamic preparation path on tDatapreRun component.

Regards

Shong

[gurn]

We are running Cloud DataPrep with tRunDataPrep calling the recipes. The dynamic schema isn't really that dynamic when it comes to completely changing the input to the DataPrep component as you need to define those schemas. I think it would be possible to do this with an on-prem install as you say here @Shicong Hong​ but not with Cloud. We have therefore built a template job that will use a different joblet for each source configured with the correct schema. This works for us and offers minimal config changes.