Qlik Use Case Reference Architecture - #6: Data Mesh Reference Architecture

[Anand_Rao]

Derive value from analytical data at scale while the data landscape, use cases, and responses constantly change.

Data Mesh creates a foundation for deriving value from analytical data at scale while the data landscape, use cases, and responses are constantly changing. This is achieved by adhering to four underpinning principles.

The first is a domain-oriented, decentralized data ownership and architecture that allows the autonomous nodes on the mesh to grow.

Next, data-as-a-product is a unit of architecture that is built, deployed, and maintained.

Third, self-service data infrastructure to enable domain teams to autonomously create and consume data products.

Last, is federated governance and interoperability standards to aggregate and correlate independent data products within the mesh. 

These principles combine to form a decentralized and distributed data mesh where domain data product owners leverage common data infrastructure via self-service, to develop pipelines that share data in a governed and open manner.

[Anand_Rao]

The architecture comprises the following components:

Data is ingested from transactional systems with low latency. Change data capture for real-time data replication ingests data without impairing production system performance.

Data Warehouse Automation accelerates the availability of analytics-ready data by automating the entire data warehouse lifecycle.

Data Lake Automation powers the process of providing continuously updated, accurate, and trusted data sets for business analytics.

Custom Transformation allows users to create flexible, fit-for-purpose data pipelines to transform raw data into data that is ready for analytics.

Data Profiling enables users to assess the quality and structure of data sources to fix data quality issues and promote good data governance.

Machine Learning enriches data with prediction, scoring, classification, and more.

Catalog & Lineage capabilities empower users to discover, govern, and protect data using AI and machine learning built on a layer of common enterprise metadata.

Analytics is used to discover, interpret, and communicate meaningful patterns in data to apply toward effective decision making

Reverse ETL replicates enriched data from the warehouse back to the operational systems of record.