Use Case Reference Architecture - #2: Data Lake Reference Architecture

[Anand_Rao]

Realize faster return on data lake investments while confidently meeting growing demands for analytics-ready data sets in real time.

Qlik Data Integration (QDI) for Data Lake Creation helps enterprises realize a faster return on their data lake investment by continuously providing accurate, timely, and trusted transactional data sets for business analytics. Unlike other solutions, QDI for Data Lakes automates the entire data pipeline from real-time data ingestion to the creation and provisioning of analytics-ready datasets, eliminating the need for manual scripting. Data engineers can now meet growing demands for analytics-ready data sets in real-time with confidence.

[Anand_Rao]

The architecture comprises the following components:

Data is ingested from transactional systems with low latency. Change data capture for real-time data replication ingests data without impairing production system performance.

Data Lake Automation powers the process of providing continuously updated, accurate, and trusted data sets for business analytics.

Custom Transformation allows users to create flexible, fit-for-purpose data pipelines to transform raw data into data that is ready for analytics.

Data Profiling enables users to assess the quality and structure of data sources to fix data quality issues and promote good data governance.

Machine Learning enriches data with prediction, scoring, classification, and more.

Catalog & Lineage capabilities empower users to discover, govern, and protect data using AI and machine learning built on a layer of common enterprise metadata.

Analytics is used to discover, interpret, and communicate meaningful patterns in data to apply toward effective decision making

Reverse ETL replicates enriched data from the warehouse back to the operational systems of record.