Talend Metadata Manager (TMM) Model vs Physical Data Model

[Anonymous]

Hi All,

I was wondering about general use cases or scenarios or recommendations where we shall be creating Model instead of Physical Data Model within Talend Metadata Manager.

Thanks,

Vivek

[Anonymous · Accepted solution]

Hi Vivek!

Yes, when harvesting metadata assets in TMM, you are asked to select either Model or Physical Data Model.

While both indicate the ability to connect to third party technologies and bring in its metadata, there are some distinct differences between the two.

A model is considered immutable (i.e., unchangeable).  It is and remains a faithful representation of what is in that third party tool - that is, the system of metadata record is the source system it is harvested from.

A physical data model (PDM) is a special kind of model and is a documentable or authorable model.  What that means is that you can extend what is in it as well as extend the information of what is in it.   So while a PDM is based on a harvested data store and starts out with the schemas, tables and columns as defined in the source system for that data store, it can be extended through TMMs data documenter to include relationships, diagrams, subject areas, logical attribution (business name and description, e.g.) of a complete data model.  This physical data model may then have a life of its own, or it may be synchronized with changes (due to harvesting new versions) to the underlying harvested model.  This makes the source of record for this kind of metadata model both the originating source system as well as TMM itself!

As you see if you check the list of available bridges on the harvest parameter configuration screen, you will see a difference in the available bridges.

Generally speaking, PDMs will generally be physical data stores (like a data base, big data cluster via Hive, etc.).  Data movement models, such as ETL or other data processes, relationship diagrams and BI models will be harvested as models.

Does this help?

Best Regards,

Cindy

[Anonymous]

Thanks for your reply.  As I understand that, we can harvest a data model as a new model or a new physical data model based on type of data store and/or data process models depending upon use case.

I had another question on the similar lines, we can drag and drop models into the glossary but not a physical data model. And as naming standards conventions apply to physical data model we cannot bootstrap a standard glossary object backed by document-able physical data model. Is this behavior of models and physical data model specific to 6.2 version.

Thanks,

Vivek

[Anonymous]

Hi Vivek,

Yes there are some limitations in the version you are currently using.  In 6.4, you can use both the Models and Physical data models to bootstrap definitions as a means to populate new terms into a Business Glossary.  As you may already know, we can also populate terms directly through the Metadata Explorer UI (directly, while mapping, or while performing other documentation on a model) or via import of a formatted csv file.

However, either way we should not conflate the bootstrap capability to create glossary terms and entries based on descriptions in a model or physical data model with the capability to apply naming standards.  TMM features a data governance based glossary feature which may contain terms with physical “abbreviations”. These terms may also be used for enforcing naming standards on physical models of databases, big data sources, etc., managed as documentable models (i.e., PDMs).

You do this by linking the glossary to the PDM and associating the category within the glossary that contains naming standards to the model on the "naming standard" tab of the parameter screen.

Best Regards,

Cindy